Compare commits

..

318 Commits

Author SHA1 Message Date
Adnan Hajdarevic 568c711625 wip 2021-04-03 18:02:40 +02:00
Adnan Hajdarević e329b6d9ff Merge pull request #518 from adhawkins/busybox-tests
Allow tests to run on systems that use busybox (such as Alpine)
2021-03-15 22:43:13 +01:00
Andy Hawkins 181672afcc Allow tests to run on systems that use busybox (such as Alpine) 2021-03-13 16:02:44 +00:00
Adnan Hajdarević d523af1b6c Fixes #497 2021-02-28 09:55:08 +01:00
Adnan Hajdarević 390e3bd772 Merge pull request #503 from TheCatLady/add-alt-docker-images
Add alternative Docker images
2021-01-29 20:08:46 +01:00
TheCatLady 21549749c0 Add alternative Docker images 2021-01-28 12:16:03 -05:00
Adnan Hajdarevic 6184509494 Add build directory to .gitignore 2021-01-26 20:53:07 +01:00
Adnan Hajdarevic b1f69564a3 Merge branch 'development' 2020-12-06 08:42:20 +01:00
Adnan Hajdarevic 159cb4a911 bump version to 2.8.0 2020-12-06 08:42:09 +01:00
Adnan Hajdarević b5af9a3968 Merge pull request #489 from moorereason/iss487-doc-string
Add string parameter example to docs
2020-12-06 08:40:31 +01:00
Adnan Hajdarević 2e4aea4cbc Merge pull request #486 from moorereason/iss439-raw-body
Add option to send raw request body to command
2020-12-06 08:39:55 +01:00
Adnan Hajdarević b6e5b11174 Merge pull request #485 from moorereason/iss234-soft-sig-errors
Add soft signature failure support
2020-12-06 08:39:31 +01:00
Adnan Hajdarević 9dec52c727 Merge pull request #484 from moorereason/iss421-slash-path
Add support for slashes in hook IDs
2020-12-06 08:38:53 +01:00
Cameron Moore f2b536dbad Add string parameter example to docs
Fixes #487
2020-12-05 16:34:49 -06:00
Cameron Moore 62f9c01cab Add option to send raw request body to command
The existing `entire-payload` option sends a JSON representation of the
parsed request body.  Add a new `raw-request-body` source to send the
raw request body.

Fixes #439
2020-11-25 10:20:10 -06:00
Cameron Moore 6d2f26d952 Add soft signature failure support
Add a new trigger-signature-soft-failures option to allow soft signature
failures in Or rules.

Fixes #234
2020-11-24 21:16:57 -06:00
Cameron Moore c2ffd465c4 Add support for slashes in hook IDs
When matching variables in routes, gorilla/mux uses a default pattern of
"[^/]+", thereby prohibiting slashes in variable matching.  Override the
default pattern to remove this restriction.

See https://github.com/gorilla/mux/blob/v1.8.0/regexp.go#L50

Fixes #421
2020-11-24 16:56:54 -06:00
Adnan Hajdarević 3e18a060ae Merge pull request #479 from moorereason/iss312-http-request
Add request source
2020-11-21 18:58:19 +01:00
Cameron Moore 6f5962f8f2 Use strings.ToLower on source name parameters 2020-11-21 10:00:03 -06:00
Cameron Moore 346c761ef6 Add request source
Add "request" source with support for "method" and "remote-addr"
parameters.  Both values are taken from the raw http.Request object.

Fixes #312
2020-11-20 16:32:55 -06:00
Adnan Hajdarević e513eb4bf4 Merge pull request #477 from moorereason/refactor-req-parsing
Move some request parsing into hook package
2020-11-19 19:44:33 +01:00
Cameron Moore 22c8a1670b Move some request parsing into hook package
Trying to simplify hookHandler.  No functional changes introduced.
2020-11-17 15:00:58 -06:00
Adnan Hajdarević 9c7f8c1ac4 Update README.md 2020-11-05 23:20:36 +01:00
Adnan Hajdarević 4fadb1171f Merge pull request #472 from moorereason/iss471-sc
Fix OrRule logic on parameter lookup failures
2020-10-26 14:09:19 +01:00
Cameron Moore dc184d2737 Fix OrRule logic on parameter lookup failures
Fixes #471
2020-10-24 11:40:27 -05:00
Adnan Hajdarević 7467933680 Merge pull request #469 from Maximization/patch-1
Add guide to the README
2020-10-14 15:31:17 +02:00
Maxim Orlov fd50118712 Add guide to the README 2020-10-13 16:12:01 +02:00
Adnan Hajdarević 67c317e741 Merge pull request #465 from moorereason/gh-action-tests
Add Github Action to build & run tests
2020-10-02 07:52:06 +02:00
Adnan Hajdarević ab3ff0343e Merge pull request #463 from moorereason/iss400
Document YAML support
2020-09-29 09:11:41 +02:00
Cameron Moore f007fa5280 Simplify build workflow 2020-09-28 21:18:38 -05:00
Cameron Moore a904537367 Add build badge to README.md 2020-09-28 20:48:39 -05:00
Cameron Moore 0814b10a16 Add Github Action to build & run tests 2020-09-28 17:20:22 -05:00
Cameron Moore d279505930 Document YAML support
Fixes #400
Updates #288
2020-09-28 14:44:21 -05:00
Adnan Hajdarević 0f4bbfac9f Merge pull request #461 from moorereason/iss289-hmac-cleanup
Transition payload hash option names to hmac
2020-09-28 04:47:24 +02:00
Cameron Moore 6bbf14f7d9 Transition payload hash option names to hmac
The payload-hash-* options are imprecisely named. Clarify their function
as HMAC validations by renaming them. The existing options will continue
to work but are deprecated.  Log a warning if the old options are used.

All tests, examples, and documentation are updated.

Fixes #289
2020-09-27 20:24:36 -05:00
Adnan Hajdarević 6797bf7cf7 Merge pull request #462 from moorereason/req-context
Add Request object to hook package to simplify API
2020-09-26 15:20:32 +02:00
Cameron Moore c6603894c1 Add Request object to hook package to simplify API
To avoid having to pass around so many parameters to the hook package,
create a Request object to store all request-specific data.  Update APIs
accordingly.
2020-09-25 19:46:06 -05:00
Adnan Hajdarević b8498c564d Merge pull request #460 from moorereason/iss456-log-exec-err
Log stdlib error on failed exec.LookPath
2020-09-25 07:13:35 +02:00
Cameron Moore dd5fa20415 Log stdlib error on failed exec.LookPath
The error returned by exec.LookPath was never surfaced to the user.
Without that detail, the user can't tell the difference between a
non-existent path and a permissions issue.

Additionally, when ExecuteCommand is an absolute path, we were still
attempting to prepend the CommandWorkingDirectory if the ExecuteCommand
was not found, which made it difficult to know which path the user
intended to execute.

This commit simplifies the logic to avoid multiple attempts with
ExecuteCommand is an absolute path and changes the error message from:

  error locating command: '/path/to/file'

to:

  error in exec: "/path/to/file": stat /path/to/file: no such file or directory
  error in exec: "/path/to/file": permission denied

Fixes #457
2020-09-24 21:02:07 -05:00
Adnan Hajdarević c7a8fbc929 Merge pull request #449 from moorereason/feature/448-return-json
Update ExtractParameterAsString to return JSON on complex types
2020-08-03 08:08:38 +02:00
Adnan Hajdarević 04ca211531 Merge pull request #446 from moorereason/feature/json-array
Add support for top-level JSON array in payload
2020-08-03 08:07:09 +02:00
Cameron Moore ae5e9e7894 Update ExtractParameterAsString to return JSON on complex types
Fixes #448
2020-07-31 11:58:12 -05:00
Adnan Hajdarević 47e5ae5527 Merge pull request #447 from moorereason/feature/docs-toc
Add a table of contents to some of the docs
2020-07-31 14:47:49 +02:00
Cameron Moore 534e99bf13 Add a table of contents to some of the docs 2020-07-29 17:23:23 -05:00
Cameron Moore 0e90ccb441 Add support for top-level JSON array in payload
Detect if leading character in JSON payload is an array bracket.  If
found, decode payload into an interface{} and then save the results into
payload["root"].  References to payload values would need to reference
the leading, "virtual" root node (i.e. "root.0.name").

Fixes #215
2020-07-29 16:56:25 -05:00
Adnan Hajdarević f692da2465 Merge pull request #445 from moorereason/bugfix/require-go14
Bugfix/require go14
2020-07-29 08:53:51 +02:00
Gabe Gałązka fb9b22a118 Change minimum golang version to 1.14 in README 2020-07-27 13:18:35 -05:00
Cameron Moore eefcd7f7d5 Require Go 1.14
When go.mod specifies go 1.14 or higher, the go tools now verify that
vendor/modules.txt is consistent with go.mod.  Fixed by running `go mod
vendor`.
2020-07-27 13:18:18 -05:00
Adnan Hajdarević c4f29b5d8b Merge pull request #432 from moorereason/bugfix/logging-gocritic
Fix issues in logging middleware
2020-05-29 09:43:32 +02:00
Adnan Hajdarević dd84a68483 Merge pull request #431 from moorereason/bugfix/pidfile-tighten
Tighten file permissions on pidfile creation
2020-05-29 09:42:55 +02:00
Cameron Moore c9199d62e4 Tighten file permissions on pidfile creation
Fixes report from gosec: "G306: Expect WriteFile permissions to be 0600
or less."  Also, use new octal number formatting.
2020-05-28 18:23:02 -05:00
Cameron Moore 3d824b47b7 Rename var to avoid shadowing bytes package
importShadow: shadow of imported package 'bytes' (gocritic)
2020-05-28 18:20:07 -05:00
Cameron Moore cc98de88ce Fix godoc comment on LogEntry.Panic 2020-05-28 18:17:41 -05:00
Adnan Hajdarević e71b45b28f Merge pull request #427 from moorereason/feature/empty-payload-signature
Warn on failed validate of empty payload signature
2020-05-23 09:28:09 +02:00
Cameron Moore 41ac427a89 Warn on failed validate of empty payload signature
If signature validation fails on an empty payload, append a note to the
end of the error message.

Updates #423
2020-05-22 14:02:12 -05:00
Adnan Hajdarević 7b3c5fd028 Merge pull request #426 from moorereason/bugfix/issue425
Fix request dumper
2020-05-22 07:15:43 +02:00
Cameron Moore 526c9a20ac Fix request dumper
The existing code had a bug in printing request params.  Simplify the
request logger by using httputil.DumpRequest.

Also print the request before handing it downstream.

Fixes #425
2020-05-21 17:47:55 -05:00
Adnan Hajdarević a75ab4f92f Merge pull request #420 from adnanh/master
Update README.md
2020-05-14 14:22:48 +02:00
Adnan Hajdarević 345bf3d409 Update README.md 2020-05-14 14:22:24 +02:00
Adnan Hajdarevic e6e324235d Bump version to v2.7.0 2020-05-12 19:14:25 +02:00
Adnan Hajdarevic 6c8d2e6b6d Merge branch 'master' into development 2020-05-12 19:13:27 +02:00
Adnan Hajdarević c8ea86f6ce Merge pull request #417 from moorereason/bugfix/error-locating-command-test
Fix missing command test
2020-05-12 13:52:37 +02:00
Cameron Moore 4f437e4642 Fix missing command test 2020-05-11 20:21:37 -05:00
Adnan Hajdarević 7267733aa8 Merge pull request #351 from dexpota/master
Add help target to Makefile
2020-04-27 21:46:23 +02:00
Adnan Hajdarević 95bd1b3072 Merge pull request #413 from moorereason/feature/go1.14ciphers
Use Go 1.14 cipher suites
2020-04-25 17:04:59 +02:00
Adnan Hajdarević 9cb199c8b3 Merge pull request #415 from moorereason/bugfix/missing-rid-on-missing-cmd
Add request ID logging on missing command
2020-04-25 16:59:21 +02:00
Cameron Moore 4407c0190b Add request ID logging on missing command 2020-04-24 15:32:33 -05:00
Cameron Moore 4897bea79f Use Go 1.14 cipher suites
Now that Go 1.14 is out, we can remove cipher_suites.go and use the
stdlib.
2020-04-24 09:13:11 -05:00
Adnan Hajdarević 38294cd0c6 Merge pull request #383 from moorereason/feature/pidfile
Add pidfile support
2020-02-14 19:35:18 +01:00
Adnan Hajdarević dc4f42bb26 Merge pull request #384 from moorereason/feature/value-walk
Show failed parameter node lookups
2020-02-14 19:13:36 +01:00
Adnan Hajdarević 472ce4863f Merge pull request #388 from moorereason/feature/multi-signature
Feature/multi signature
2020-01-07 10:50:41 +01:00
Wyatt Johnson de626ab2bb fix: updated based on review
- added support for sha512
- added notes to docs
2020-01-06 18:23:30 -06:00
Wyatt Johnson f8c8932866 fix: spelling 2020-01-06 18:23:20 -06:00
Wyatt Johnson 6d3b81fc61 fix: simplify implementation 2020-01-06 18:23:09 -06:00
Wyatt Johnson 11e0031a9f feat: added multiple sig support 2020-01-06 18:22:55 -06:00
Adnan Hajdarević 53f63a7614 Merge pull request #387 from adnanh/revert-355-master
Revert "Multiple Signature Support"
2020-01-03 23:40:05 +01:00
Adnan Hajdarević 8c5b2e0c17 Revert "Multiple Signature Support" 2020-01-03 23:38:49 +01:00
Cameron Moore 7fa3a8900c Show failed parameter node lookups
When attempting to match a JSON path for initial setup, it would be
helpful to know where the path failed. This change logs the failed
parameter node. For example, if you are trying to match path "a.b.d.e",
but you failed to include the "c" node, webhook will log an error
"parameter node not found: d.e" to assist in troubleshooting.
2019-12-30 21:51:11 -06:00
Cameron Moore e1634fe669 Add missing windows dependency 2019-12-29 18:08:35 -06:00
Cameron Moore 876c853073 Add pidfile support
Copy a simple implementation from the Moby project, since importing
their package would pull in too many dependencies.

Fixes #320
2019-12-29 18:00:55 -06:00
Adnan Hajdarević 569921cd72 Merge pull request #381 from moorereason/feature/logfile
Feature/logfile
2019-12-29 20:10:16 +01:00
Cameron Moore fda328dc23 Refactor fatal logging during service startup
Create a log queue to postpone the first log write until after
privilege dropping and log file opening.
2019-12-28 20:50:33 -06:00
Adnan Hajdarević aa03daeff8 Merge pull request #380 from moorereason/feature/http-methods
Feature/http methods
2019-12-28 14:17:39 +01:00
Cameron Moore 811481298a Fix method not allowed log message 2019-12-28 07:09:36 -06:00
Cameron Moore 5af6e4d1ec Open listener port earlier 2019-12-27 12:01:12 -06:00
Cameron Moore 725fda68dc Add logfile feature 2019-12-27 11:51:44 -06:00
Cameron Moore 157f468e0c Refactor cli HTTP methods behavior
The CLI HTTP methods option now sets the default allowed methods while
allowing an individual hook definition to override the default.
2019-12-27 11:22:04 -06:00
Cameron Moore e1249a9ddb Add global HTTP methods to starting log message 2019-12-26 15:17:01 -06:00
Cameron Moore a03e812615 Update HTTP methods to sanitize user input 2019-12-26 14:54:27 -06:00
Cameron Moore c38778ba62 Add HTTP methods cli parameter
Allows to globally restrict HTTP methods.

Fixes #248
2019-12-26 14:54:27 -06:00
Cameron Moore 3414f34025 Add per-hook HTTP method restrictions 2019-12-26 14:54:01 -06:00
Adnan Hajdarević 66562fdb41 Merge pull request #379 from moorereason/feature/drop-privs
Add setuid and setgid options for dropping privileges
2019-12-26 20:48:07 +01:00
Cameron Moore 77159d9db6 Add setuid & setgid options
Only applicable on unix systems, although Go doesn't support Linux at
this time.
2019-12-26 10:30:31 -06:00
Cameron Moore 35d1cedc24 Rewrite server to use explicit listener 2019-12-26 10:29:14 -06:00
Adnan Hajdarević f38dfbbf78 Merge pull request #373 from moorereason/feature/multipart
Add multipart form data support
2019-12-26 12:46:31 +01:00
Adnan Hajdarević 78b0610218 Merge pull request #378 from moorereason/feature/sighup
Add SIGHUP support
2019-12-26 12:46:12 +01:00
Cameron Moore c6c270c7dd Add SIGHUP support
Fixes #352
2019-12-25 14:10:57 -06:00
Cameron Moore 1c779a0d75 Update multipart form data logic
All form values are simply added to the payload map without processing.
JSON parsing of values happens later.
2019-12-25 09:08:23 -06:00
Cameron Moore 8702b37430 Add multipart form data examples 2019-12-25 09:05:15 -06:00
Cameron Moore 5b4e60e7d7 Add multipart form data section to README 2019-12-25 09:05:15 -06:00
Cameron Moore 93632d077c Add multipart form data support 2019-12-25 09:05:15 -06:00
Adnan Hajdarević cc5cbae14f Merge pull request #376 from moorereason/feature/xml
Add XML payload support
2019-12-25 08:32:09 +01:00
Cameron Moore 779ff0ad10 Fix XML error message 2019-12-24 19:47:21 -06:00
Adnan Hajdarević d8bd2662ff Merge pull request #375 from moorereason/feature/gorilla-only
Use gorilla/mux for middleware and extend
2019-12-24 23:21:01 +01:00
Cameron Moore 28e0012470 Update XML docs 2019-12-24 16:09:55 -06:00
Cameron Moore 3463804a7c Add XML payload support
Fixes #238
2019-12-24 15:58:49 -06:00
Cameron Moore 3f4520da67 Require Go 1.13+ 2019-12-24 14:56:01 -06:00
Cameron Moore be815d0a41 Use gorilla/mux for middleware and extend
- Use gorilla/mux for middleware.
- Add Dumper, RequestID, and Logger middlewares.
- Add makeURL helper
2019-12-24 11:57:26 -06:00
Adnan Hajdarević 93ce24d3f3 Merge pull request #371 from moorereason/feature/internal-hook
Make hook package internal
2019-12-23 09:34:48 +01:00
Adnan Hajdarević e72a7d2e22 Merge pull request #355 from wyattjoh/master
Multiple Signature Support
2019-12-22 22:31:13 +01:00
Cameron Moore 40d9dcd6d4 Make hook package internal
The hook package API is not meant for public consumption.
2019-12-21 11:55:42 -06:00
Adnan Hajdarević c872aae7e8 Merge pull request #369 from moorereason/feature/gofrs-uuid
Use gofrs/uuid instead of satori/go.uuid
2019-12-19 07:52:43 +01:00
Cameron Moore 8ff3848ea3 Use gofrs/uuid instead of satori/go.uuid
The satori package appears to be unmaintained.  The gofrs package is a
fork that is actively maintained by a larger group of Go developers.
2019-12-18 21:17:13 -06:00
Adnan Hajdarević 7b87d6092f Merge pull request #368 from moorereason/feature/go-mod
Use Go modules
2019-12-18 10:52:56 +01:00
Cameron Moore 669414ca70 Use Go modules
Fixes #367
2019-12-17 12:34:59 -06:00
Wyatt Johnson 3f5fee20c0 fix: updated based on review
- added support for sha512
- added notes to docs
2019-12-17 10:18:08 -07:00
Wyatt Johnson c6e809a1a2 fix: spelling 2019-12-17 10:05:18 -07:00
Wyatt Johnson 2088f61cba fix: simplify implementation 2019-12-17 10:05:18 -07:00
Wyatt Johnson a818e29113 feat: added multiple sig support 2019-12-17 10:05:18 -07:00
Adnan Hajdarevic 8fe6c9a05d Update version to 2.6.11 2019-12-15 14:30:54 +01:00
Adnan Hajdarević 7c4e6e94fc Merge pull request #365 from moorereason/feature/364-constant-time
Use constant time string compare for match value
2019-12-11 07:06:12 +01:00
Cameron Moore 31e76bcd00 Use constant time string compare for match value
Fixes #364
2019-12-10 22:22:13 -06:00
Adnan Hajdarević c47c06e822 Merge pull request #363 from moorereason/feature/arm64-travisci
Add arm64 to Travis CI
2019-12-10 00:23:38 +01:00
Cameron Moore bf3d042da6 Use master instead of tip 2019-12-09 16:57:55 -06:00
Cameron Moore d05911cdcb Add arm64 to Travis CI 2019-12-09 16:50:19 -06:00
Adnan Hajdarević 634ca84807 Merge pull request #362 from moorereason/feature/cipher-suites
Feature/cipher suites
2019-12-09 21:52:25 +01:00
Cameron Moore 8c46a8343b Document minimum Go release 2019-12-05 17:01:38 -06:00
Cameron Moore 13d5630e80 Update docs for TLS version and cipher suite options 2019-12-03 21:36:14 -06:00
Cameron Moore f1003560f1 Add list cipher suites support 2019-12-03 21:35:16 -06:00
Cameron Moore 997db04b9f Require Go 1.12 or newer
Go 1.11 does not support TLS 1.3.  To simplify cipher suite selection,
we now require at least Go 1.12.
2019-12-03 21:31:23 -06:00
Cameron Moore 769e743563 Add missing files 2019-12-03 15:21:18 -06:00
Cameron Moore 43f519a712 Add TLS version and cipher suites options
Default to TLS 1.2 and secure cipher suites.

Built for Go 1.13. Code in cipher_suites.go taken from Go tip commit
0ee22d9, which is scheduled for the upcoming Go 1.14 release.  Once Go
1.14 is released, we can remove this file and use the stdlib.

Fixes #244
2019-12-03 15:13:12 -06:00
Adnan Hajdarević a617b1a6ac Merge pull request #361 from adnanh/feature/check-payload-hash-sha512
Add SHA512 payload check rule
2019-12-02 22:34:07 +01:00
Adnan Hajdarević 9117f4f6d6 Merge pull request #360 from adnanh/improvement/content-type-based-payload-parsing
Fix invalid assumption that multipart forms can be parsed in te same way as urlencoded forms.
2019-12-02 22:33:45 +01:00
Adnan Hajdarevic b53996f175 Add tests for SHA512 payload hash check rule. 2019-12-02 19:49:56 +01:00
Adnan Hajdarevic 154177e46a Add documentation for SHA512 payload hash check rule. 2019-12-02 19:49:34 +01:00
Adnan Hajdarevic d4e98281d7 Add SHA512 payload check rule. 2019-12-02 19:48:59 +01:00
Adnan Hajdarevic ce186487f4 Format the file using go fmt. 2019-12-02 19:03:38 +01:00
Adnan Hajdarevic 1110f82443 Add test for unsupported content type error message. 2019-12-02 19:01:20 +01:00
Adnan Hajdarevic a99abd4e6f Fix invalid assumption in code that multipart forms can be parsed in the same way as urlencoded forms.
Refactored code to use switch-case statement over the `Content-Type` header and log unsupported content types instead of silently failing.
Also made the `x-www-form-urlencoded` content type handler more specific (as opposed to the previous code which looked for `form` occurence in the value),
as we need to use different logic for multipart forms, which we'll hopefully implement soon.

The issue with multipart forms that we have to handle first is that the files are being written to temporary files, and as such, for async hooks
webhook cannot guarantee they'll be available after we close the request; that, and the fact that we don't have code that will properly serialize
and pass such Golang objects to the script, as there are several fields which might be interesting to the end user.
2019-12-02 18:49:24 +01:00
Fabrizio Destro 8728ec4786 Add help target to Makefile 2019-10-19 23:16:17 +02:00
Adnan Hajdarevic 34ae132930 Bump webhook version to 2.6.10 2019-09-24 19:45:40 +02:00
Adnan Hajdarevic f993aaa11c Merge branch 'master' into development 2019-09-24 19:35:31 +02:00
Adnan Hajdarević d82e838554 Merge pull request #342 from moorereason/doc-content-type
Document incoming-payload-content-type hook setting
2019-09-21 11:50:46 +02:00
Cameron Moore 9c35aa070c Document incoming-payload-content-type hook setting
As implemented in PR #206
2019-09-20 08:50:44 -05:00
Adnan Hajdarević 14ee68a06e Create FUNDING.yml 2019-09-20 01:48:22 +02:00
Adnan Hajdarević e0e1dd8ade Delete appveyor.yml 2019-09-20 01:42:52 +02:00
Adnan Hajdarević 9852f0f0a5 Merge pull request #341 from moorereason/wintrav
Add Windows to Travis CI
2019-09-20 01:41:01 +02:00
Cameron Moore 2cf3f4e1a8 Add Windows to Travis CI 2019-09-19 08:42:33 -05:00
Adnan Hajdarević 1cf531b1c3 Merge pull request #339 from moorereason/StatusOK
Fix TestWebhook tests
2019-09-19 10:52:44 +02:00
Cameron Moore 74e55e3089 Update go versions in Travis CI
Test against the latest three minor releases.
2019-09-18 17:00:22 -05:00
Cameron Moore 6c77ff0a2c Fix TestWebhook tests
PR #266 appears to have changed the default response code to StatusOK.
waitForServerReady() was expected a StatusNotFound response, which was
preventing all TestWebhook tests from running.
2019-09-18 16:59:11 -05:00
Adnan Hajdarević 002c332b68 Merge pull request #327 from dexpota/master
Fix issue with relative paths and command execution
2019-09-18 18:07:07 +02:00
Adnan Hajdarević ffba396523 Merge pull request #337 from moorereason/iss333
Update GetParameter to support keys with dots
2019-09-18 18:02:25 +02:00
Cameron Moore b016e99ea6 Update GetParameter to support keys with dots
Fixes #333
2019-09-13 13:10:14 -05:00
Cameron Moore e47f9afb11 Fix failing ip-whitelist tests 2019-09-13 13:09:44 -05:00
Fabrizio Destro d3fd9bddd9 Fix issue with relative paths and command execution 2019-08-25 20:08:39 +02:00
Adnan Hajdarević 42b72b028b Merge pull request #319 from Htbaa/issue-313
Replaced fmt.Frpintf calls with fmt.Fprint when there's no formatting…
2019-06-21 22:04:12 +02:00
Christiaan Kras c6939d57dc Replaced fmt.Frpintf calls with fmt.Fprint when there's no formatting used
This fixes #313
2019-04-24 14:59:38 +02:00
Adnan Hajdarević 90f751a61d Merge pull request #266 from aioobe/development
Added SuccessHttpResponseCode hook setting
2019-04-17 18:13:25 +02:00
Adnan Hajdarević e86c2cf610 Merge branch 'development' into development 2019-04-17 18:11:12 +02:00
Adnan Hajdarević 0aa7395e21 Merge pull request #304 from johnpmitsch/travis
Add travis CI webhook example
2019-02-13 09:53:03 +01:00
John Mitsch 3f834f5c3d Add travis webhook example 2019-02-12 20:51:29 -05:00
Adnan Hajdarević 5bca86cdb2 Add snap store link to the README.md 2019-02-12 10:29:39 +01:00
Adnan Hajdarević 896d1608ca Merge pull request #297 from moorereason/iss207
Return errors on empty secrets during signature validations
2019-01-08 09:30:08 +01:00
Adnan Hajdarević e5c18aa87e Merge pull request #298 from moorereason/iss290
Allow multiple values for ip-whitelist
2019-01-08 09:29:32 +01:00
Cameron Moore f056f94305 Allow multiple values for ip-whitelist
Allow the value of ip-whitelist to consist of multiple space-separated
addresses or CIDRs.

Updates #290
2019-01-02 16:50:23 -06:00
Cameron Moore 1a17dc83fe Return errors on empty secrets during signature validations
Fixes #207
2019-01-02 16:09:27 -06:00
Adnan Hajdarević 753734428f Merge pull request #283 from adnanh/healthcheck-route
Add `/` route handler to return 200 OK which can be used as a healthcheck endpoint
2018-11-17 19:03:20 +01:00
Adnan Hajdarevic f76426e9b0 add handler for the route to be used as a healtcheck endpoint, fixes #233 2018-11-17 19:01:26 +01:00
Adnan Hajdarević 5803d5e849 Merge pull request #282 from adnanh/master
Backmerge
2018-11-16 18:59:25 +01:00
Adnan Hajdarević 385898b21f Merge pull request #281 from ruliezz/patch-1
Update docs with a Gitea example
2018-11-16 18:58:58 +01:00
ruliezz ce7f8d5d28 Updated with a Gitea example 2018-11-16 15:23:39 +01:00
Adnan Hajdarević b2899d1d3e Update README.md 2018-11-16 10:46:35 +01:00
Adnan Hajdarević e8628cd662 Add more guides to the README 2018-11-16 10:45:37 +01:00
Adnan Hajdarević 4e1719d966 Merge pull request #278 from adnanh/add-exe-extension-to-windows-build
Fix Makefile to include .exe extension for windows builds
2018-11-13 21:27:24 +01:00
Adnan Hajdarevic 98f86cf044 Fix Makefile to include .exe extension for windows builds 2018-11-13 21:12:20 +01:00
Adnan Hajdarevic fc0544e4a2 Bump version to 2.6.9 2018-11-13 21:01:42 +01:00
Adnan Hajdarević 537f5c21bc Merge pull request #277 from ZachCheung/master
docs: fix link
2018-11-13 18:03:14 +01:00
Zach Cheung a0880ab82d docs: fix link 2018-11-13 23:30:47 +08:00
Adnan Hajdarević 5636ead921 Merge pull request #269 from alyssais/hook-def-links
Fix links in Hook Definition docs
2018-10-05 21:37:46 +02:00
Alyssa Ross 01e0c9e972 Fix links in Hook Definition docs 2018-10-05 11:24:36 +01:00
Andreas Lundblad 54a7190113 Forgot a rename in previous refactoring. 2018-09-17 20:41:51 +02:00
Andreas Lundblad b65bdbbb24 Removed trailing tab 2018-09-17 20:35:51 +02:00
Andreas Lundblad ef3f43f89f Added SuccessHttpResponseCode handling for case when capture output is set to true. 2018-09-15 16:06:18 +02:00
Andreas Lundblad 22073d8847 Renamed http-response-code to success-http-response-code 2018-09-15 16:00:42 +02:00
Andreas Lundblad c05ca8c528 Added HttpResponseCode hook setting 2018-09-15 15:55:28 +02:00
Adnan Hajdarević f9e799fea0 Merge pull request #206 from dcj/feature/incoming-payload-content-type
added support for incoming-payload-content-type
2018-09-14 11:51:06 +02:00
Adnan Hajdarević 9b99452b60 Merge pull request #256 from vkovalchuk/master
Fixed links to other .md files in Hook-Definition.md doc
2018-09-14 11:43:31 +02:00
Adnan Hajdarević b93cdc346e Update README.md 2018-08-02 11:29:49 +02:00
Adnan Hajdarević d59f6228ad Merge pull request #262 from Awea/master
Add an entry to guide list
2018-07-23 10:09:42 +02:00
Awea 78c8c61bf2 📝 Update README 2018-07-23 09:54:52 +02:00
Vladimir Kovalchuk 7ed5d4af9b Fixed links to other .md files in Hook-Definition.md doc 2018-06-13 17:32:46 +03:00
Adnan Hajdarević 7905c74687 Merge pull request #237 from kirecek/fix/links-in-docs
Add .md  suffix for links to related doc pages in "Hook-Definition"
2018-06-05 09:38:19 +02:00
Adnan Hajdarević 2fb08ab579 Merge pull request #255 from 464bb26bac556e85b6fd6b524347b103/patch-1
Include shebang info from wiki/home.md in readme
2018-06-05 09:29:43 +02:00
md5(donics) 681e8b6459 Include shebang info wiki/home.md in readme
A change was made in response to #60 (fork/exec: exec format error) on the wiki that would be useful to also propagate to the readme.
2018-06-04 14:44:04 -04:00
Erik Jankovic 8a3770db29 fix: links for related doc pages
Signed-off-by: Erik Jankovic <erik.jankovic@vnet.eu>
2018-03-01 15:27:36 +01:00
Adnan Hajdarević ae54669c02 Merge pull request #228 from moorereason/iss225
Fix some tests for Windows
2018-02-19 11:00:34 +01:00
Adnan Hajdarević b449793825 Merge pull request #229 from moorereason/tidyup
Minor Housecleaning PR
2018-02-19 11:00:17 +01:00
Cameron Moore 66a9e48e39 Fix unnecessary nil check around range 2018-02-16 20:36:42 -06:00
Cameron Moore d85ee5e068 Use strings.TrimPrefix 2018-02-16 20:33:17 -06:00
Cameron Moore 7da4d8ba9f Use strings.Contains 2018-02-16 20:31:23 -06:00
Cameron Moore 8d260c6a7e Apply gofmt 2018-02-16 20:26:33 -06:00
Cameron Moore 48061f1508 Simplify boolean some comparisons 2018-02-16 20:23:25 -06:00
Cameron Moore cfed5cfe4b Fix unnecessary use of printf 2018-02-16 20:18:40 -06:00
Cameron Moore 471c849c50 Fix another race condition in TestWebhook
There's the potential for a race condition where we try to read the logs
buffer before the logs have been flushed by the webhook process. Kill
the process to flush the logs before testing against the log buffer.
2018-02-16 14:36:08 -06:00
Cameron Moore 337621998e Fix race in TestWebhook
Previous commit misused a bytes.Buffer. Protect the buffer with a
mutex.
2018-02-15 19:53:28 -06:00
Cameron Moore 0feeb945fc Fix some tests for Windows
This commit incorporates some tests into the main TestWebhook framework.  New features to TestWebhook:

- Check log output against Regexp
- Add Testing sub-tests

Updates #225
2018-02-15 19:20:39 -06:00
Adnan Hajdarević 4f9ed434c5 Merge pull request #227 from moorereason/iss226
Add Travis CI and Appveyor configurations
2018-02-15 20:24:46 +01:00
Cameron Moore 0934b9414c Add Travis CI and Appveyor configurations
Fixes #226
2018-02-14 16:35:54 -06:00
Adnan Hajdarević 356870358d Merge pull request #218 from hassanbabaie/development
Document updates for new scalr-signature feature
2018-02-14 15:39:42 +01:00
Hass_SEA 6dc331726d Updated Examples document with scalr-signature
Updated the Examples document with an example of how you would use the scalr-signature match rule
2018-01-18 12:43:12 -08:00
Hass_SEA 3f8dbf09dc Correct typos - Rules Document with scalr-signature
Correct typos - Rules Document with scalr-signature
2018-01-18 12:40:05 -08:00
Hass_SEA dcda096b5d Update Rules Document with scalr-signature
Updated Rules Document with scalr-signature information
2018-01-18 12:36:16 -08:00
Hass_SEA 7079128eca Merge pull request #1 from adnanh/development
Merge latest Adnanh/webhook Development into fork
2018-01-18 11:42:28 -08:00
Adnan Hajdarević 6e3ec89ce1 Merge pull request #210 from hassanbabaie/master
Add support for Scalr webhook signature verification (new Match Rule) #200 - Updated
2018-01-16 09:26:26 +01:00
Adnan Hajdarević 10396a5434 Update README.md 2018-01-11 10:34:55 +01:00
Adnan Hajdarević d009919755 Update README.md 2018-01-11 10:34:07 +01:00
Adnan Hajdarevic a811db410b check before removing 2017-12-21 13:25:19 +01:00
Adnan Hajdarević 357c471667 Merge pull request #212 from adnanh/fix-file-panic
Fix nilpointer dereference when file cannot be created
2017-12-21 13:15:18 +01:00
Adnan Hajdarevic 85889fe378 Fix nilpointer dereference when file cannot be created 2017-12-21 13:14:07 +01:00
Hass_SEA b595694658 Update to support Scalr Signature Verification
Add a new match rule type that checks for a Scalr webhook signature. Tracking ticket #200

The signature algorithm is described here:
https://scalr-wiki.atlassian.net/wiki/spaces/docs/pages/6193247/Webhook+Security+and+Authentication

An example match rule ifor a Scalr webhook will look like:

"match": {
"type": "scalr-signature",
"secret": ""
}
2017-12-19 12:48:10 -08:00
Adnan Hajdarevic ecbba514e5 Bump version 2017-12-13 20:46:15 +01:00
Adnan Hajdarevic 13555e1ef1 Merge 2017-12-13 20:45:22 +01:00
Adnan Hajdarević ddec431ae9 Update README.md 2017-11-29 11:17:48 +01:00
Donald Clark Jackson f84edae99d added support for incoming-payload-content-type 2017-11-27 14:02:57 -08:00
Adnan Hajdarević 514388e39e Merge pull request #166 from gdubicki/add_CaptureCommandOutputOnError
Add option to capture output of failed commands
2017-11-13 19:24:53 +01:00
Adnan Hajdarević a49364ae93 Update README.md 2017-11-13 08:44:31 +01:00
Adnan Hajdarević 6c4e52d87e Update README.md 2017-11-13 08:39:33 +01:00
Adnan Hajdarević b161de13f2 Add DigitalOcean to sponsors 2017-11-13 08:38:55 +01:00
Greg Dubicki e2f6e4eb37 Add tests for capturing command output
and fix running tests on macOS, where there is no
/bin/true...
2017-11-11 21:40:34 +01:00
Greg Dubicki 0d3d29055b Allow hookecho to exit with codes other than 0 2017-11-11 21:05:36 +01:00
Greg Dubicki a40fba5e29 Add CaptureCommandOutputOnError
to include stdout & stderror in failed executions,
with docs.
2017-11-11 18:46:55 +01:00
Adnan Hajdarević 137ded093d Update README.md 2017-11-11 12:39:18 +01:00
Adnan Hajdarevic 24ec77cb6a crisp 2017-11-11 12:29:39 +01:00
Adnan Hajdarevic 7f3529e302 Add project logo 2017-11-11 12:18:20 +01:00
Adnan Hajdarević df781925f1 Merge pull request #195 from moorereason/iss193
Add Template docs
2017-11-11 01:33:48 +01:00
Cameron Moore 44d19e34a0 Add Template docs
In addition to the Template docs, README is updated to use the "w" URL
shortcut.

Fixes #193
2017-11-10 16:11:29 -06:00
Adnan Hajdarević b1e0c0bf01 Merge pull request #194 from moorereason/iss192
Add docs
2017-11-10 21:05:50 +01:00
Cameron Moore bab32db017 Add docs 2017-11-10 12:54:15 -06:00
Cameron Moore 5a69eab2b3 Convert README.md to unix line endings 2017-11-10 12:48:19 -06:00
Adnan Hajdarević 01aa178bb4 Merge pull request #161 from moorereason/templates
Allow hooks file to be parsed as a template
2017-11-09 12:00:14 +01:00
Cameron Moore f5f04ddaa2 Allow hooks file to be parsed as a template
Add a -template command line option that instructs webhook to parse the
hooks files as Go text templates.

Includes a `getenv` template func for retrieving environment variables.
2017-11-08 09:14:58 -06:00
Adnan Hajdarević ba0adb117a Merge pull request #174 from col-panic/master
Pass "big" binary files #162
2017-11-08 10:56:49 +01:00
Marco Descher c107bb4539 Merge branch 'master' of https://github.com/adnanh/webhook into colpanic/master 2017-11-08 08:33:22 +01:00
Adnan Hajdarevic 36c5a52539 Merge branch 'master' of github.com:adnanh/webhook 2017-11-07 19:22:41 +01:00
Adnan Hajdarevic 027f2d7363 Fix tests 2017-11-07 19:22:32 +01:00
Marco Descher c9abc252e8 Delete hook files by default 2017-11-07 16:38:12 +01:00
Marco Descher 6145634670 Adapt to #173 2017-11-07 16:19:22 +01:00
Marco Descher 41b75acc70 Merge branch 'master' into master 2017-11-07 16:14:47 +01:00
Adnan Hajdarević 83e688efd3 Update README.md 2017-11-05 16:54:05 +01:00
Adnan Hajdarević b1851784b5 Merge pull request #188 from adnanh/development
Merge pull request #187 from adnanh/master
2017-11-05 16:50:31 +01:00
Adnan Hajdarević 3c0508a8d3 Merge pull request #187 from adnanh/master
Backmerge
2017-11-05 16:50:09 +01:00
Adnan Hajdarević 7a91168f23 Update README.md 2017-11-05 16:49:36 +01:00
Adnan Hajdarević 2adf724295 Update README.md 2017-11-05 16:23:10 +01:00
Adnan Hajdarević 2f7eb31e8d Update README.md 2017-11-05 16:09:31 +01:00
Adnan Hajdarević 717ad779a3 Merge pull request #185 from cavneb/patch-2
Remove Code Sponsor
2017-11-05 16:08:27 +01:00
Eric Berry 1e67afc3de Remove Code Sponsor 2017-11-05 06:14:08 -08:00
Adnan Hajdarević c094740905 Update README.md 2017-11-04 21:52:26 +01:00
Adnan Hajdarević de494d4cc6 Add guides featuring webhook 2017-11-04 21:00:33 +01:00
Adnan Hajdarević 65bd1cdd10 Update webhook.go 2017-11-04 20:49:57 +01:00
Adnan Hajdarevic 7af1fe8e02 Merge branch 'master' into development 2017-11-04 20:48:51 +01:00
Adnan Hajdarevic 5def341367 Vendor 2017-11-04 20:31:12 +01:00
Adnan Hajdarević a422170c61 Update README.md 2017-11-04 19:13:27 +01:00
Adnan Hajdarević 739388c0dc Update README.md 2017-10-26 08:33:01 +02:00
Adnan Hajdarević 9c1efc91a2 Merge pull request #180 from stblassitude/master
Vendor code using godep.
2017-10-17 11:50:43 +02:00
Stefan Bethke b663783717 Vendor code using godep.
For 3rd parties building binary packages, and for build consistency in
general, it is very helpful to have the same set of dependencies at
any time the product is built.

See [tools/godep](https://github.com/tools/godep) for further details.
2017-10-09 20:05:32 +02:00
Adnan Hajdarević e748910b6a Merge pull request #176 from opencollective/opencollective
Activating Open Collective
2017-09-20 14:34:59 +02:00
Aseem Sood 78ca57908c Added backers and sponsors on the README 2017-09-19 17:54:43 -04:00
Aseem Sood 3f42ac2906 Added .github/ISSUE_TEMPLATE.md (optional) 2017-09-19 17:54:19 -04:00
Aseem Sood 91c1675cc0 Added CONTRIBUTING.md (optional) 2017-09-19 17:54:19 -04:00
Marco Descher 213e4529e8 #162 do use temporary files, provide env variable 2017-09-15 13:30:44 +02:00
Adnan Hajdarevic 8c96ffd5f9 use /bin/true instead of /usr/bin/true 2017-09-14 20:50:59 +02:00
Adnan Hajdarević f0e26bc03c Merge pull request #164 from ivanpesin/static-parameter-warning
Static parameter warning
2017-09-14 19:44:16 +02:00
Adnan Hajdarević 71269d8b45 Merge pull request #173 from ivanpesin/log-request-id
Add request-id to verbose log messages
2017-09-14 19:34:47 +02:00
Ivan Pesin da39872a77 Add request-id to verbose log messages 2017-09-12 23:12:12 -05:00
Adnan Hajdarević 985d5197ed Merge pull request #172 from ivanpesin/fsnotify-vim
Fix to support hot reload with vim editor changes
2017-09-12 10:23:22 +02:00
Ivan Pesin 142e912d6a Fix to support hot reload with vim editor changes 2017-09-12 00:10:16 -05:00
Ivan Pesin 241061c26d Improved TestStaticParams to check for log ouput 2017-09-10 21:08:15 -05:00
Ivan Pesin c6febd35b5 Added a test for command static parameters handling 2017-09-10 19:35:08 -05:00
Marco Descher 78aa85e0c1 Revert to original hook import 2017-08-30 14:23:29 +02:00
Marco Descher 395fb41d23 Provide means to transfer files #162 add deleteOnExit 2017-08-29 14:42:14 +02:00
Ivan Pesin 1fc4445668 Produce warnings if unable to locate binary and if static parameters specified erroneously 2017-08-25 23:31:02 -04:00
Ivan Pesin d52d7bde1c Fixed source code formatting with go fmt (spaces to tabs) 2017-08-25 23:30:08 -04:00
Marco Descher 34c4b1c166 Provide means to transfer files #162 2017-08-25 15:01:57 +02:00
Adnan Hajdarević 147c95dd8b Merge pull request #158 from adnanh/development
2.6.5
2017-08-09 09:53:37 +02:00
Adnan Hajdarevic cfd138c96d Fix #141 2017-08-09 09:52:59 +02:00
Adnan Hajdarević c19e514ee9 Merge pull request #143 from wrouesnel/yaml-decoding
Use the github.com/ghodss/yaml library to unserialize webhooks.
2017-08-09 09:29:58 +02:00
Adnan Hajdarević c7ec25f378 Merge pull request #156 from andrewharvey/patch-1
Update README.md for Debian stretch release of webhook package
2017-08-02 14:06:24 +02:00
Andrew Harvey 6916f89a2b Update README.md for Debian stretch release of webhook package 2017-08-02 22:02:55 +10:00
Adnan Hajdarević ed251c3cc7 Merge pull request #149 from adnanh/master
Readme backmerge
2017-07-06 09:25:12 +02:00
Adnan Hajdarević 9db13ef649 Update README.md 2017-07-06 09:24:50 +02:00
Adnan Hajdarević 98512463d8 Update README.md 2017-07-06 09:24:00 +02:00
Will Rouesnel 53ec2e7acc Add tests for YAML file parsing. 2017-06-29 02:34:58 +10:00
Adnan Hajdarević c581e0532b Merge pull request #144 from ilkelma/use-combined-output
Change to CombinedOutput to consume Stderr as well
2017-06-16 10:59:03 +02:00
ilkelma 3a75bbf4db Change to CombinedOutput to consume Stderr as well 2017-06-13 16:23:59 -04:00
Will Rouesnel 2f1ea098c0 Use the github.com/ghodss/yaml library to unserialize webhooks.
This supports both JSON and YAML seamlessly, providing for an easier human
parseable format on disk.
2017-06-12 19:56:09 +10:00
Adnan Hajdarević ccd28908db Create README.md 2017-06-08 11:16:41 +02:00
Adnan Hajdarević 69a4aec1f3 Merge pull request #139 from adnanh/development
2.6.4
2017-05-27 17:15:40 +02:00
Adnan Hajdarevic a38875213a fix tests, log ip of the incoming request, bump version 2017-05-27 17:14:38 +02:00
Adnan Hajdarević 249962f002 Merge pull request #127 from adnanh/master
Backmerge
2017-04-10 16:08:47 +02:00
Adnan Hajdarević ced9e01895 Update README.md 2017-04-10 16:08:14 +02:00
Adnan Hajdarević 5561fa3308 Merge pull request #126 from carsonip/patch-1
Add prebuilt binaries to readme.md
2017-04-09 10:37:52 +02:00
Carson Ip 47b82c7c4a Add prebuilt binaries to readme.md
Link to prebuilt binaries should be available in readme.md for users who don't have Go installed and just want things to work.
2017-04-09 01:55:59 +02:00
Adnan Hajdarević 3978b6687d Merge pull request #125 from adnanh/development
webhook 2.6.3
2017-04-07 11:00:37 +02:00
Adnan Hajdarevic 81b1bd7c7b bump up the version to 2.6.3 2017-04-07 10:59:40 +02:00
Adnan Hajdarevic 45cf312b8d Merge branch 'master' into development 2017-04-07 10:59:15 +02:00
Adnan Hajdarević 2cfc1ce2ff Merge pull request #124 from raphink/sha256
Add sha256 payload check
2017-04-07 10:55:16 +02:00
Raphaël Pinson 6b1021925b Add sha256 payload check 2017-04-06 17:49:31 +02:00
Adnan Hajdarević af22498d1e Update README.md 2017-04-06 17:01:08 +02:00
Adnan Hajdarević 5b567d1631 Update README.md 2017-04-06 11:09:17 +02:00
Adnan Hajdarević 4aa81b727f Merge pull request #116 from handcraftedbits/development
Add IP whitelist match rule
2017-02-21 12:50:48 +01:00
HandcraftedBits 173273b466 Add IP whitelist match rule. 2017-02-20 22:00:39 -05:00
Adnan Hajdarević 8803239e6d Merge pull request #115 from adnanh/development
Support loading hooks from multiple files
2017-02-11 23:01:01 +01:00
Adnan Hajdarevic c8a83349d2 remove hooks after the underlying file is removed 2017-02-11 22:58:49 +01:00
Adnan Hajdarevic 1da40d4634 Merge branch 'master' into development 2017-02-11 12:15:04 +01:00
Adnan Hajdarevic 79453b7a22 Use hooks.json as default file if no hooks file have been specified 2016-12-05 21:28:38 +01:00
Adnan Hajdarevic 8207c6cf12 Started work on multi file hooks loading 2016-12-05 21:22:34 +01:00
540 changed files with 215316 additions and 1275 deletions
+3
View File
@@ -0,0 +1,3 @@
# These are supported funding model platforms
open_collective: webhook
+2
View File
@@ -0,0 +1,2 @@
<!-- Love webhook? Please consider supporting our collective:
👉 https://opencollective.com/webhook/donate -->
+23
View File
@@ -0,0 +1,23 @@
name: build
on: [push, pull_request]
jobs:
build:
strategy:
matrix:
go-version: [1.14.x, 1.15.x]
os: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go-version }}
id: go
- name: Build
run: go build -v
- name: Test
run: go test -v ./...
+2
View File
@@ -2,3 +2,5 @@
.cover
coverage
webhook
/test/hookecho
build
+32
View File
@@ -0,0 +1,32 @@
language: go
go:
- 1.14.x
- master
os:
- linux
- osx
- windows
arch:
- amd64
- arm64
matrix:
fast_finish: true
allow_failures:
- go: master
exclude:
- os: windows
go: master
- os: windows
arch: arm64
- os: osx
arch: arm64
install:
- go get -d -v -t ./...
script:
- go test -v -race ./...
+67
View File
@@ -0,0 +1,67 @@
# Contribute
## Introduction
First, thank you for considering contributing to webhook! It's people like you that make the open source community such a great community! 😊
We welcome any type of contribution, not only code. You can help with
- **QA**: file bug reports, the more details you can give the better (e.g. screenshots with the console open)
- **Marketing**: writing blog posts, howto's, printing stickers, ...
- **Community**: presenting the project at meetups, organizing a dedicated meetup for the local community, ...
- **Code**: take a look at the [open issues](issues). Even if you can't write code, commenting on them, showing that you care about a given issue matters. It helps us triage them.
- **Money**: we welcome financial contributions in full transparency on our [open collective](https://opencollective.com/webhook).
## Your First Contribution
Working on your first Pull Request? You can learn how from this *free* series, [How to Contribute to an Open Source Project on GitHub](https://egghead.io/series/how-to-contribute-to-an-open-source-project-on-github).
## Submitting code
Any code change should be submitted as a pull request. The description should explain what the code does and give steps to execute it. The pull request should also contain tests.
## Code review process
The bigger the pull request, the longer it will take to review and merge. Try to break down large pull requests in smaller chunks that are easier to review and merge.
It is also always helpful to have some context for your pull request. What was the purpose? Why does it matter to you?
## Financial contributions
We also welcome financial contributions in full transparency on our [open collective](https://opencollective.com/webhook).
Anyone can file an expense. If the expense makes sense for the development of the community, it will be "merged" in the ledger of our open collective by the core contributors and the person who filed the expense will be reimbursed.
## Questions
If you have any questions, create an [issue](issue) (protip: do a quick search first to see if someone else didn't ask the same question before!).
You can also reach us at hello@webhook.opencollective.com.
## Credits
### Contributors
Thank you to all the people who have already contributed to webhook!
<a href="graphs/contributors"><img src="https://opencollective.com/webhook/contributors.svg?width=890" /></a>
### Backers
Thank you to all our backers! [[Become a backer](https://opencollective.com/webhook#backer)]
<a href="https://opencollective.com/webhook#backers" target="_blank"><img src="https://opencollective.com/webhook/backers.svg?width=890"></a>
### Sponsors
Thank you to all our sponsors! (please ask your company to also support this open source project by [becoming a sponsor](https://opencollective.com/webhook#sponsor))
<a href="https://opencollective.com/webhook/sponsor/0/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/0/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/1/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/1/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/2/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/2/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/3/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/3/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/4/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/4/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/5/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/5/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/6/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/6/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/7/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/7/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/8/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/8/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/9/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/9/avatar.svg"></a>
<!-- This `CONTRIBUTING.md` is based on @nayafia's template https://github.com/nayafia/contributing-template -->
+22 -7
View File
@@ -1,12 +1,18 @@
OS = darwin freebsd linux openbsd windows
OS = darwin freebsd linux openbsd
ARCHS = 386 arm amd64 arm64
all: build release
.DEFAULT_GOAL := help
build: deps
.PHONY: help
help:
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-16s\033[0m %s\n", $$1, $$2}'
all: build release release-windows
build: deps ## Build the project
go build
release: clean deps
release: clean deps ## Generate releases for unix systems
@for arch in $(ARCHS);\
do \
for os in $(OS);\
@@ -18,12 +24,21 @@ release: clean deps
done \
done
test: deps
release-windows: clean deps ## Generate release for windows
@for arch in $(ARCHS);\
do \
echo "Building windows-$$arch"; \
mkdir -p build/webhook-windows-$$arch/; \
GOOS=windows GOARCH=$$arch go build -o build/webhook-windows-$$arch/webhook.exe; \
tar cz -C build -f build/webhook-windows-$$arch.tar.gz webhook-windows-$$arch; \
done
test: deps ## Execute tests
go test ./...
deps:
deps: ## Install dependencies using go get
go get -d -v -t ./...
clean:
clean: ## Remove building artifacts
rm -rf build
rm -f webhook
+140 -29
View File
@@ -1,12 +1,14 @@
[![ghit.me](https://ghit.me/badge.svg?repo=adnanh/webhook)](https://ghit.me/repo/adnanh/webhook) [![Join the chat at https://gitter.im/adnanh/webhook](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/adnanh/webhook?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Flattr this](https://button.flattr.com/flattr-badge-large.png)](https://flattr.com/submit/auto?user_id=adnanh&url=https%3A%2F%2Fwww.github.com%2Fadnanh%2Fwebhook) [Donate via PayPal](https://paypal.me/hookdoo) | [Patreon page](https://www.patreon.com/webhook)
# What is webhook?
[webhook](https://github.com/adnanh/webhook/) is a lightweight configurable tool written in Go, that allows you to easily create HTTP endpoints (hooks) on your server, which you can use to execute configured commands. You can also pass data from the HTTP request (such as headers, payload or query variables) to your commands. [webhook](https://github.com/adnanh/webhook/) also allows you to specify rules which have to be satisfied in order for the hook to be triggered.
# What is webhook? ![build-status][badge]
For example, if you're using Github or Bitbucket, you can use [webhook](https://github.com/adnanh/webhook/) to set up a hook that runs a redeploy script for your project on your staging server, whenever you push changes to the master branch of your project.
<img src="https://github.com/adnanh/webhook/raw/development/docs/logo/logo-128x128.png" alt="Webhook" align="left" />
[webhook][w] is a lightweight configurable tool written in Go, that allows you to easily create HTTP endpoints (hooks) on your server, which you can use to execute configured commands. You can also pass data from the HTTP request (such as headers, payload or query variables) to your commands. [webhook][w] also allows you to specify rules which have to be satisfied in order for the hook to be triggered.
If you use Slack, you can set up an "Outgoing webhook integration" to run various commands on your server, which can then report back directly to your Slack channels using the "Incoming webhook integrations".
For example, if you're using Github or Bitbucket, you can use [webhook][w] to set up a hook that runs a redeploy script for your project on your staging server, whenever you push changes to the master branch of your project.
[webhook](https://github.com/adnanh/webhook/) aims to do nothing more than it should do, and that is:
If you use Mattermost or Slack, you can set up an "Outgoing webhook integration" or "Slash command" to run various commands on your server, which can then report back directly to you or your channels using the "Incoming webhook integrations", or the appropriate response body.
[webhook][w] aims to do nothing more than it should do, and that is:
1. receive the request,
2. parse the headers, payload and query variables,
3. check if the specified rules for the hook are satisfied,
@@ -15,25 +17,43 @@ If you use Slack, you can set up an "Outgoing webhook integration" to run variou
Everything else is the responsibility of the command's author.
---
# Hookdoo
<a href="https://www.hookdoo.com/?github"><img src="https://www.hookdoo.com/logo/logo.svg" height="96" alt="hookdoo" align="left" /></a>
If you don't have time to waste configuring, hosting, debugging and maintaining your webhook instance, we offer a __SaaS__ solution that has all of the capabilities webhook provides, plus a lot more, and all that packaged in a nice friendly web interface. If you are interested, find out more at [hookdoo website](https://www.hookdoo.com/?ref=github-webhook-readme). If you have any questions, you can contact us at info@hookdoo.com
#
<a href="https://www.hookdeck.io/?ref=adnanh-webhook"><img src="http://hajdarevic.net/hookdeck-logo.svg" height="17" alt="hookdeck" align="left" /></a> If you need a way of inspecting, monitoring and replaying webhooks without the back and forth troubleshooting, [give Hookdeck a try!](https://www.hookdeck.io/?ref=adnanh-webhook)
# Getting started
## Installation
### Building from source
To get started, first make sure you've properly set up your [Golang](http://golang.org/doc/install) environment and then run the
To get started, first make sure you've properly set up your [Go](http://golang.org/doc/install) 1.14 or newer environment and then run
```bash
$ go get github.com/adnanh/webhook
$ go build github.com/adnanh/webhook
```
to get the latest version of the [webhook](https://github.com/adnanh/webhook/).
to build the latest version of the [webhook][w].
### Using package manager
#### Debian "sid"
If you are using unstable version of Debian linux ("sid"), you can install webhook using `apt-get install webhook` which will install community packaged version (thanks [@freeekanayaka](https://github.com/freeekanayaka)) from https://packages.debian.org/sid/webhook
#### Snap store
[![Get it from the Snap Store](https://snapcraft.io/static/images/badges/en/snap-store-white.svg)](https://snapcraft.io/webhook)
#### Ubuntu
If you are using Ubuntu linux (17.04 or later), you can install webhook using `sudo apt-get install webhook` which will install community packaged version.
#### Debian
If you are using Debian linux ("stretch" or later), you can install webhook using `sudo apt-get install webhook` which will install community packaged version (thanks [@freeekanayaka](https://github.com/freeekanayaka)) from https://packages.debian.org/sid/webhook
### Download prebuilt binaries
Prebuilt binaries for different architectures are available at [GitHub Releases](https://github.com/adnanh/webhook/releases).
## Configuration
Next step is to define some hooks you want [webhook](https://github.com/adnanh/webhook/) to serve. Begin by creating an empty file named `hooks.json`. This file will contain an array of hooks the [webhook](https://github.com/adnanh/webhook/) will serve. Check [Hook definition page](https://github.com/adnanh/webhook/wiki/Hook-Definition) to see the detailed description of what properties a hook can contain, and how to use them.
Next step is to define some hooks you want [webhook][w] to serve.
[webhook][w] supports JSON or YAML configuration files, but we'll focus primarily on JSON in the following example.
Begin by creating an empty file named `hooks.json`. This file will contain an array of hooks the [webhook][w] will serve. Check [Hook definition page](docs/Hook-Definition.md) to see the detailed description of what properties a hook can contain, and how to use them.
Let's define a simple hook named `redeploy-webhook` that will run a redeploy script located in `/var/scripts/redeploy.sh`.
Let's define a simple hook named `redeploy-webhook` that will run a redeploy script located in `/var/scripts/redeploy.sh`. Make sure that your bash script has `#!/bin/sh` shebang on top.
Our `hooks.json` file will now look like this:
```json
@@ -46,7 +66,14 @@ Our `hooks.json` file will now look like this:
]
```
You can now run [webhook](https://github.com/adnanh/webhook/) using
**NOTE:** If you prefer YAML, the equivalent `hooks.yaml` file would be:
```yaml
- id: redeploy-webhook
execute-command: "/var/scripts/redeploy.sh"
command-working-directory: "/var/webhook"
```
You can now run [webhook][w] using
```bash
$ /path/to/webhook -hooks hooks.json -verbose
```
@@ -56,32 +83,115 @@ It will start up on default port 9000 and will provide you with one HTTP endpoin
http://yourserver:9000/hooks/redeploy-webhook
```
Check [webhook parameters page](https://github.com/adnanh/webhook/wiki/Webhook-Parameters) to see how to override the ip, port and other settings such as hook hotreload, verbose output, etc, when starting the [webhook](https://github.com/adnanh/webhook/).
Check [webhook parameters page](docs/Webhook-Parameters.md) to see how to override the ip, port and other settings such as hook hotreload, verbose output, etc, when starting the [webhook][w].
By performing a simple HTTP GET or POST request to that endpoint, your specified redeploy script would be executed. Neat!
However, hook defined like that could pose a security threat to your system, because anyone who knows your endpoint, can send a request and execute your command. To prevent that, you can use the `"trigger-rule"` property for your hook, to specify the exact circumstances under which the hook would be triggered. For example, you can use them to add a secret that you must supply as a parameter in order to successfully trigger the hook. Please check out the [Hook rules page](https://github.com/adnanh/webhook/wiki/Hook-Rules) for detailed list of available rules and their usage.
However, hook defined like that could pose a security threat to your system, because anyone who knows your endpoint, can send a request and execute your command. To prevent that, you can use the `"trigger-rule"` property for your hook, to specify the exact circumstances under which the hook would be triggered. For example, you can use them to add a secret that you must supply as a parameter in order to successfully trigger the hook. Please check out the [Hook rules page](docs/Hook-Rules.md) for detailed list of available rules and their usage.
# Using HTTPS
[webhook](https://github.com/adnanh/webhook/) by default serves hooks using http. If you want [webhook](https://github.com/adnanh/webhook/) to serve secure content using https, you can use the `-secure` flag while starting [webhook](https://github.com/adnanh/webhook/). Files containing a certificate and matching private key for the server must be provided using the `-cert /path/to/cert.pem` and `-key /path/to/key.pem` flags. If the certificate is signed by a certificate authority, the cert file should be the concatenation of the server's certificate followed by the CA's certificate.
## Multipart Form Data
[webhook][w] provides limited support the parsing of multipart form data.
Multipart form data can contain two types of parts: values and files.
All form _values_ are automatically added to the `payload` scope.
Use the `parse-parameters-as-json` settings to parse a given value as JSON.
All files are ignored unless they match one of the following criteria:
# CORS Headers
If you want to set CORS headers, you can use the `-header name=value` flag while starting [webhook](https://github.com/adnanh/webhook/) to set the appropriate CORS headers that will be returned with each response.
1. The `Content-Type` header is `application/json`.
1. The part is named in the `parse-parameters-as-json` setting.
# Interested in running webhook inside of a Docker container?
You can use [almir/webhook](https://hub.docker.com/r/almir/webhook/) docker image, or create your own (please read [this discussion](https://github.com/adnanh/webhook/issues/63)).
In either case, the given file part will be parsed as JSON and added to the `payload` map.
# Examples
Check out [Hook examples page](https://github.com/adnanh/webhook/wiki/Hook-Examples) for more complex examples of hooks.
## Templates
[webhook][w] can parse the hooks configuration file as a Go template when given the `-template` [CLI parameter](docs/Webhook-Parameters.md). See the [Templates page](docs/Templates.md) for more details on template usage.
# Contributing
Any form of contribution is welcome and highly appreciated.
## Using HTTPS
[webhook][w] by default serves hooks using http. If you want [webhook][w] to serve secure content using https, you can use the `-secure` flag while starting [webhook][w]. Files containing a certificate and matching private key for the server must be provided using the `-cert /path/to/cert.pem` and `-key /path/to/key.pem` flags. If the certificate is signed by a certificate authority, the cert file should be the concatenation of the server's certificate followed by the CA's certificate.
Big thanks to [all the current contributors](https://github.com/adnanh/webhook/graphs/contributors) for their contributions!
TLS version and cipher suite selection flags are available from the command line. To list available cipher suites, use the `-list-cipher-suites` flag. The `-tls-min-version` flag can be used with `-list-cipher-suites`.
# Community Contributions
## CORS Headers
If you want to set CORS headers, you can use the `-header name=value` flag while starting [webhook][w] to set the appropriate CORS headers that will be returned with each response.
## Interested in running webhook inside of a Docker container?
You can use one of the following Docker images, or create your own (please read [this discussion](https://github.com/adnanh/webhook/issues/63)):
- [almir/webhook](https://github.com/almir/docker-webhook)
- [roxedus/webhook](https://github.com/Roxedus/docker-webhook)
- [thecatlady/webhook](https://github.com/thecatlady/docker-webhook)
## Examples
Check out [Hook examples page](docs/Hook-Examples.md) for more complex examples of hooks.
### Guides featuring webhook
- [Plex 2 Telegram](https://gitlab.com/-/snippets/1972594) by [@psyhomb](https://github.com/psyhomb)
- [Webhook & JIRA](https://sites.google.com/site/mrxpalmeiras/notes/jira-webhooks) by [@perfecto25](https://github.com/perfecto25)
- [Trigger Ansible AWX job runs on SCM (e.g. git) commit](http://jpmens.net/2017/10/23/trigger-awx-job-runs-on-scm-commit/) by [@jpmens](http://mens.de/)
- [Deploy using GitHub webhooks](https://davidauthier.wearemd.com/blog/deploy-using-github-webhooks.html) by [@awea](https://davidauthier.wearemd.com)
- [Setting up Automatic Deployment and Builds Using Webhooks](https://willbrowning.me/setting-up-automatic-deployment-and-builds-using-webhooks/) by [Will Browning](https://willbrowning.me/about/)
- [Auto deploy your Node.js app on push to GitHub in 3 simple steps](https://webhookrelay.com/blog/2018/07/17/auto-deploy-on-git-push/) by Karolis Rusenas
- [Automate Static Site Deployments with Salt, Git, and Webhooks](https://www.linode.com/docs/applications/configuration-management/automate-a-static-site-deployment-with-salt/) by [Linode](https://www.linode.com)
- [Using Prometheus to Automatically Scale WebLogic Clusters on Kubernetes](https://blogs.oracle.com/weblogicserver/using-prometheus-to-automatically-scale-weblogic-clusters-on-kubernetes-v5) by [Marina Kogan](https://blogs.oracle.com/author/9a4fe754-1cc2-4c64-95fc-360642b62927)
- [Github Pages and Jekyll - A New Platform for LACNIC Labs](https://labs.lacnic.net/a-new-platform-for-lacniclabs/) by [Carlos Martínez Cagnazzo](https://twitter.com/carlosm3011)
- [How to Deploy React Apps Using Webhooks and Integrating Slack on Ubuntu](https://www.alibabacloud.com/blog/how-to-deploy-react-apps-using-webhooks-and-integrating-slack-on-ubuntu_594116) by Arslan Ud Din Shafiq
- [Private webhooks](https://ihateithe.re/2018/01/private-webhooks/) by [Thomas](https://ihateithe.re/colophon/)
- [Adventures in webhooks](https://medium.com/@draketech/adventures-in-webhooks-2d6584501c62) by [Drake](https://medium.com/@draketech)
- [GitHub pro tips](http://notes.spencerlyon.com/2016/01/04/github-pro-tips/) by [Spencer Lyon](http://notes.spencerlyon.com/)
- [XiaoMi Vacuum + Amazon Button = Dash Cleaning](https://www.instructables.com/id/XiaoMi-Vacuum-Amazon-Button-Dash-Cleaning/) by [c0mmensal](https://www.instructables.com/member/c0mmensal/)
- [Set up Automated Deployments From Github With Webhook](https://maximorlov.com/automated-deployments-from-github-with-webhook/) by [Maxim Orlov](https://twitter.com/_maximization)
- VIDEO: [Gitlab CI/CD configuration using Docker and adnanh/webhook to deploy on VPS - Tutorial #1](https://www.youtube.com/watch?v=Qhn-lXjyrZA&feature=youtu.be) by [Yes! Let's Learn Software Engineering](https://www.youtube.com/channel/UCH4XJf2BZ_52fbf8fOBMF3w)
- ...
- Want to add your own? Open an Issue or create a PR :-)
## Community Contributions
See the [webhook-contrib][wc] repository for a collections of tools and helpers related to [webhook][w] that have been contributed by the [webhook][w] community.
## Need help?
Check out [existing issues](https://github.com/adnanh/webhook/issues) to see if someone else also had the same problem, or [open a new one](https://github.com/adnanh/webhook/issues/new).
# Support active development
## Sponsors
## <a href="https://www.digitalocean.com/?ref=webhook"><img src="http://www.hajdarevic.net/DO_Logo_Horizontal_Blue.png" alt="DigitalOcean" width="250"/></a>
[DigitalOcean](https://www.digitalocean.com/?ref=webhook) is a simple and robust cloud computing platform, designed for developers.
## <a href="https://www.browserstack.com/?ref=webhook"><img src="http://www.hajdarevic.net/browserstack.svg" alt="BrowserStack" width="250"/></a>
[BrowserStack](https://www.browserstack.com/?ref=webhook) is a cloud-based cross-browser testing tool that enables developers to test their websites across various browsers on different operating systems and mobile devices, without requiring users to install virtual machines, devices or emulators.
---
Support this project by becoming a sponsor. Your logo will show up here with a link to your website.
<a href="https://opencollective.com/webhook/sponsor/0/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/0/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/1/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/1/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/2/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/2/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/3/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/3/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/4/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/4/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/5/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/5/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/6/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/6/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/7/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/7/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/8/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/8/avatar.svg"></a>
<a href="https://opencollective.com/webhook/sponsor/9/website" target="_blank"><img src="https://opencollective.com/webhook/sponsor/9/avatar.svg"></a>
## By contributing
This project exists thanks to all the people who contribute. [Contribute!](CONTRIBUTING.md).
<a href="graphs/contributors"><img src="https://opencollective.com/webhook/contributors.svg?width=890" /></a>
## By giving money
- [OpenCollective Backer](https://opencollective.com/webhook#backer)
- [OpenCollective Sponsor](https://opencollective.com/webhook#sponsor)
- [PayPal](https://paypal.me/hookdoo)
- [Patreon](https://www.patreon.com/webhook)
- [Faircode](https://faircode.io/product/webhook?utm_source=badge&utm_medium=badgelarge&utm_campaign=webhook)
- [Flattr](https://flattr.com/submit/auto?user_id=adnanh&url=https%3A%2F%2Fwww.github.com%2Fadnanh%2Fwebhook)
---
Thank you to all our backers!
<a href="https://opencollective.com/webhook#backers" target="_blank"><img src="https://opencollective.com/webhook/backers.svg?width=890"></a>
# License
The MIT License (MIT)
@@ -109,3 +219,4 @@ THE SOFTWARE.
[w]: https://github.com/adnanh/webhook
[wc]: https://github.com/adnanh/webhook-contrib
[badge]: https://github.com/adnanh/webhook/workflows/build/badge.svg
Binary file not shown.
+28
View File
@@ -0,0 +1,28 @@
# Hook definition
Hooks are defined as objects in the JSON or YAML hooks configuration file. Please note that in order to be considered valid, a hook object must contain the `id` and `execute-command` properties. All other properties are considered optional.
## Properties (keys)
* `id` - specifies the ID of your hook. This value is used to create the HTTP endpoint (http://yourserver:port/hooks/your-hook-id)
* `execute-command` - specifies the command that should be executed when the hook is triggered
* `command-working-directory` - specifies the working directory that will be used for the script when it's executed
* `response-message` - specifies the string that will be returned to the hook initiator
* `response-headers` - specifies the list of headers in format `{"name": "X-Example-Header", "value": "it works"}` that will be returned in HTTP response for the hook
* `success-http-response-code` - specifies the HTTP status code to be returned upon success
* `incoming-payload-content-type` - sets the `Content-Type` of the incoming HTTP request (ie. `application/json`); useful when the request lacks a `Content-Type` or sends an erroneous value
* `http-methods` - a list of allowed HTTP methods, such as `POST` and `GET`
* `include-command-output-in-response` - boolean whether webhook should wait for the command to finish and return the raw output as a response to the hook initiator. If the command fails to execute or encounters any errors while executing the response will result in 500 Internal Server Error HTTP status code, otherwise the 200 OK status code will be returned.
* `include-command-output-in-response-on-error` - boolean whether webhook should include command stdout & stderror as a response in failed executions. It only works if `include-command-output-in-response` is set to `true`.
* `parse-parameters-as-json` - specifies the list of arguments that contain JSON strings. These parameters will be decoded by webhook and you can access them like regular objects in rules and `pass-arguments-to-command`.
* `pass-arguments-to-command` - specifies the list of arguments that will be passed to the command. Check [Referencing request values page](Referencing-Request-Values.md) to see how to reference the values from the request. If you want to pass a static string value to your command you can specify it as
`{ "source": "string", "name": "argumentvalue" }`
* `pass-environment-to-command` - specifies the list of arguments that will be passed to the command as environment variables. If you do not specify the `"envname"` field in the referenced value, the hook will be in format "HOOK_argumentname", otherwise "envname" field will be used as it's name. Check [Referencing request values page](Referencing-Request-Values.md) to see how to reference the values from the request. If you want to pass a static string value to your command you can specify it as
`{ "source": "string", "envname": "SOMETHING", "name": "argumentvalue" }`
* `pass-file-to-command` - specifies a list of entries that will be serialized as a file. Incoming [data](Referencing-Request-Values.md) will be serialized in a request-temporary-file (otherwise parallel calls of the hook would lead to concurrent overwritings of the file). The filename to be addressed within the subsequent script is provided via an environment variable. Use `envname` to specify the name of the environment variable. If `envname` is not provided `HOOK_` and the name used to reference the request value are used. Defining `command-working-directory` will store the file relative to this location, if not provided, the systems temporary file directory will be used. If `base64decode` is true, the incoming binary data will be base 64 decoded prior to storing it into the file. By default the corresponding file will be removed after the webhook exited.
* `trigger-rule` - specifies the rule that will be evaluated in order to determine should the hook be triggered. Check [Hook rules page](Hook-Rules.md) to see the list of valid rules and their usage
* `trigger-rule-mismatch-http-response-code` - specifies the HTTP status code to be returned when the trigger rule is not satisfied
* `trigger-signature-soft-failures` - allow signature validation failures within Or rules; by default, signature failures are treated as errors.
## Examples
Check out [Hook examples page](Hook-Examples.md) for more complex examples of hooks.
+623
View File
@@ -0,0 +1,623 @@
# Hook Examples
Hooks are defined in a hooks configuration file in either JSON or YAML format,
although the examples on this page all use the JSON format.
🌱 This page is still a work in progress. Feel free to contribute!
### Table of Contents
* [Incoming Github webhook](#incoming-github-webhook)
* [Incoming Bitbucket webhook](#incoming-bitbucket-webhook)
* [Incoming Gitlab webhook](#incoming-gitlab-webhook)
* [Incoming Gogs webhook](#incoming-gogs-webhook)
* [Incoming Gitea webhook](#incoming-gitea-webhook)
* [Slack slash command](#slack-slash-command)
* [A simple webhook with a secret key in GET query](#a-simple-webhook-with-a-secret-key-in-get-query)
* [JIRA Webhooks](#jira-webhooks)
* [Pass File-to-command sample](#pass-file-to-command-sample)
* [Incoming Scalr Webhook](#incoming-scalr-webhook)
* [Travis CI webhook](#travis-ci-webhook)
* [XML Payload](#xml-payload)
* [Multipart Form Data](#multipart-form-data)
* [Pass string arguments to command](#pass-string-arguments-to-command)
## Incoming Github webhook
```json
[
{
"id": "webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "head_commit.id"
},
{
"source": "payload",
"name": "pusher.name"
},
{
"source": "payload",
"name": "pusher.email"
}
],
"trigger-rule":
{
"and":
[
{
"match":
{
"type": "payload-hmac-sha1",
"secret": "mysecret",
"parameter":
{
"source": "header",
"name": "X-Hub-Signature"
}
}
},
{
"match":
{
"type": "value",
"value": "refs/heads/master",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
]
}
}
]
```
## Incoming Bitbucket webhook
Bitbucket does not pass any secrets back to the webhook. [Per their documentation](https://confluence.atlassian.com/bitbucket/manage-webhooks-735643732.html#Managewebhooks-trigger_webhookTriggeringwebhooks), in order to verify that the webhook came from Bitbucket you must whitelist the IP range `104.192.143.0/24`:
```json
[
{
"id": "webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "actor.username"
}
],
"trigger-rule":
{
"match":
{
"type": "ip-whitelist",
"ip-range": "104.192.143.0/24"
}
}
}
]
```
## Incoming Gitlab Webhook
Gitlab provides webhooks for many kinds of events.
Refer to this URL for example request body content: [gitlab-ce/integrations/webhooks](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/project/integrations/webhooks.md)
Values in the request body can be accessed in the command or to the match rule by referencing 'payload' as the source:
```json
[
{
"id": "redeploy-webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "user_name"
}
],
"response-message": "Executing redeploy script",
"trigger-rule":
{
"match":
{
"type": "value",
"value": "<YOUR-GENERATED-TOKEN>",
"parameter":
{
"source": "header",
"name": "X-Gitlab-Token"
}
}
}
}
]
```
## Incoming Gogs webhook
```json
[
{
"id": "webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "head_commit.id"
},
{
"source": "payload",
"name": "pusher.name"
},
{
"source": "payload",
"name": "pusher.email"
}
],
"trigger-rule":
{
"and":
[
{
"match":
{
"type": "payload-hmac-sha256",
"secret": "mysecret",
"parameter":
{
"source": "header",
"name": "X-Gogs-Signature"
}
}
},
{
"match":
{
"type": "value",
"value": "refs/heads/master",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
]
}
}
]
```
## Incoming Gitea webhook
```json
[
{
"id": "webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "head_commit.id"
},
{
"source": "payload",
"name": "pusher.name"
},
{
"source": "payload",
"name": "pusher.email"
}
],
"trigger-rule":
{
"and":
[
{
"match":
{
"type": "value",
"value": "mysecret",
"parameter":
{
"source": "payload",
"name": "secret"
}
}
},
{
"match":
{
"type": "value",
"value": "refs/heads/master",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
]
}
}
]
```
## Slack slash command
```json
[
{
"id": "redeploy-webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"response-message": "Executing redeploy script",
"trigger-rule":
{
"match":
{
"type": "value",
"value": "<YOUR-GENERATED-TOKEN>",
"parameter":
{
"source": "payload",
"name": "token"
}
}
}
}
]
```
## A simple webhook with a secret key in GET query
__Not recommended in production due to low security__
`example.com:9000/hooks/simple-one` - won't work
`example.com:9000/hooks/simple-one?token=42` - will work
```json
[
{
"id": "simple-one",
"execute-command": "/path/to/command.sh",
"response-message": "Executing simple webhook...",
"trigger-rule":
{
"match":
{
"type": "value",
"value": "42",
"parameter":
{
"source": "url",
"name": "token"
}
}
}
}
]
```
## JIRA Webhooks
[Guide by @perfecto25](https://sites.google.com/site/mrxpalmeiras/notes/jira-webhooks)
## Pass File-to-command sample
### Webhook configuration
<pre>
[
{
"id": "test-file-webhook",
"execute-command": "/bin/ls",
"command-working-directory": "/tmp",
"pass-file-to-command":
[
{
"source": "payload",
"name": "binary",
"envname": "ENV_VARIABLE", // to use $ENV_VARIABLE in execute-command
// if not defined, $HOOK_BINARY will be provided
"base64decode": true, // defaults to false
}
],
"include-command-output-in-response": true
}
]
</pre>
### Sample client usage
Store the following file as `testRequest.json`.
<pre>
{"binary":"iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAA2lpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuMC1jMDYwIDYxLjEzNDc3NywgMjAxMC8wMi8xMi0xNzozMjowMCAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wUmlnaHRzPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvcmlnaHRzLyIgeG1sbnM6eG1wTU09Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9tbS8iIHhtbG5zOnN0UmVmPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvc1R5cGUvUmVzb3VyY2VSZWYjIiB4bWxuczp4bXA9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8iIHhtcFJpZ2h0czpNYXJrZWQ9IkZhbHNlIiB4bXBNTTpEb2N1bWVudElEPSJ4bXAuZGlkOjEzMTA4RDI0QzMxQjExRTBCMzYzRjY1QUQ1Njc4QzFBIiB4bXBNTTpJbnN0YW5jZUlEPSJ4bXAuaWlkOjEzMTA4RDIzQzMxQjExRTBCMzYzRjY1QUQ1Njc4QzFBIiB4bXA6Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDUzMgV2luZG93cyI+IDx4bXBNTTpEZXJpdmVkRnJvbSBzdFJlZjppbnN0YW5jZUlEPSJ1dWlkOkFDMUYyRTgzMzI0QURGMTFBQUI4QzUzOTBEODVCNUIzIiBzdFJlZjpkb2N1bWVudElEPSJ1dWlkOkM5RDM0OTY2NEEzQ0REMTFCMDhBQkJCQ0ZGMTcyMTU2Ii8+IDwvcmRmOkRlc2NyaXB0aW9uPiA8L3JkZjpSREY+IDwveDp4bXBtZXRhPiA8P3hwYWNrZXQgZW5kPSJyIj8+IBFgEwAAAmJJREFUeNqkk89rE1EQx2d/NNq0xcYYayPYJDWC9ODBsKIgAREjBmvEg2cvHnr05KHQ9iB49SL+/BMEfxBQKHgwCEbTNNIYaqgaoanFJi+rcXezye4689jYkIMIDnx47837zrx583YFx3Hgf0xA6/dJyAkkgUy4vgryAnmNWH9L4EVmotFoKplMHgoGg6PkrFarjXQ6/bFcLj/G5W1E+3NaX4KZeDx+dX5+7kg4HBlmrC6JoiDFYrGhROLM/mp1Y6JSqdCd3/SW0GUqEAjkl5ZyHTSHKBQKnO6a9khD2m5cr91IJBJ1VVWdiM/n6LruNJtNDs3JR3ukIW03SHTHi8iVsbG9I51OG1bW16HVasHQZopDc/JZVgdIQ1o3BmTkEnJXURS/KIpgGAYPkCQJPi0u8uzDKQN0XQPbtgE1MmrHs9nsfSqAEjxCNtHxZHLy4G4smUQgyzL4LzOegDGGp1ucVqsNqKVrpJCM7F4hg6iaZvhqtZrg8XjA4xnAU3XeKLqWaRImoIZeQXVjQO5pYp4xNVirsR1erxer2O4yfa227WCwhtWoJmn7m0h270NxmemFW4706zMm8GCgxBGEASCfhnukIW03iFdQnOPz0LNKp3362JqQzSw4u2LXBe+Bs3xD+/oc1NxN55RiC9fOme0LEQiRf2rBzaKEeJJ37ZWTVunBeGN2WmQjg/DeLTVP89nzAive2dMwlo9bpFVC2xWMZr+A720FVn88fAUb3wDMOjyN7YNc6TvUSHQ4AH6TOUdLL7em68UtWPsJqxgTpgeiLu1EBt1R+Me/mF7CQPTfAgwAGxY2vOTrR3oAAAAASUVORK5CYII="}
</pre>
use then the curl tool to execute a request to the webhook.
<pre>
#!/bin/bash
curl -H "Content-Type:application/json" -X POST -d @testRequest.json \
http://localhost:9000/hooks/test-file-webhook
</pre>
or in a single line, using https://github.com/jpmens/jo to generate the JSON code
<pre>
jo binary=%filename.zip | curl -H "Content-Type:application/json" -X POST -d @- \
http://localhost:9000/hooks/test-file-webhook
</pre>
## Incoming Scalr Webhook
[Guide by @hassanbabaie]
Scalr makes webhook calls based on an event to a configured webhook endpoint (for example Host Down, Host Up). Webhook endpoints are URLs where Scalr will deliver Webhook notifications.
Scalr assigns a unique signing key for every configured webhook endpoint.
Refer to this URL for information on how to setup the webhook call on the Scalr side: [Scalr Wiki Webhooks](https://scalr-wiki.atlassian.net/wiki/spaces/docs/pages/6193173/Webhooks)
In order to leverage the Signing Key for addtional authentication/security you must configure the trigger rule with a match type of "scalr-signature".
```json
[
{
"id": "redeploy-webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"include-command-output-in-response": true,
"trigger-rule":
{
"match":
{
"type": "scalr-signature",
"secret": "Scalr-provided signing key"
}
},
"pass-environment-to-command":
[
{
"envname": "EVENT_NAME",
"source": "payload",
"name": "eventName"
},
{
"envname": "SERVER_HOSTNAME",
"source": "payload",
"name": "data.SCALR_SERVER_HOSTNAME"
}
]
}
]
```
## Travis CI webhook
Travis sends webhooks as `payload=<JSON_STRING>`, so the payload needs to be parsed as JSON. Here is an example to run on successful builds of the master branch.
```json
[
{
"id": "deploy",
"execute-command": "/root/my-server/deployment.sh",
"command-working-directory": "/root/my-server",
"parse-parameters-as-json": [
{
"source": "payload",
"name": "payload"
}
],
"trigger-rule":
{
"and":
[
{
"match":
{
"type": "value",
"value": "passed",
"parameter": {
"name": "payload.state",
"source": "payload"
}
}
},
{
"match":
{
"type": "value",
"value": "master",
"parameter": {
"name": "payload.branch",
"source": "payload"
}
}
}
]
}
}
]
```
## JSON Array Payload
If the JSON payload is an array instead of an object, `webhook` will process the payload and place it into a "root" object.
Therefore, references to payload values must begin with `root.`.
For example, given the following payload (taken from the Sendgrid Event Webhook documentation):
```json
[
{
"email": "example@test.com",
"timestamp": 1513299569,
"smtp-id": "<14c5d75ce93.dfd.64b469@ismtpd-555>",
"event": "processed",
"category": "cat facts",
"sg_event_id": "sg_event_id",
"sg_message_id": "sg_message_id"
},
{
"email": "example@test.com",
"timestamp": 1513299569,
"smtp-id": "<14c5d75ce93.dfd.64b469@ismtpd-555>",
"event": "deferred",
"category": "cat facts",
"sg_event_id": "sg_event_id",
"sg_message_id": "sg_message_id",
"response": "400 try again later",
"attempt": "5"
}
]
```
A reference to the second item in the array would look like this:
```json
[
{
"id": "sendgrid",
"execute-command": "{{ .Hookecho }}",
"trigger-rule": {
"match": {
"type": "value",
"parameter": {
"source": "payload",
"name": "root.1.event"
},
"value": "deferred"
}
}
}
]
```
## XML Payload
Given the following payload:
```xml
<app>
<users>
<user id="1" name="Jeff" />
<user id="2" name="Sally" />
</users>
<messages>
<message id="1" from_user="1" to_user="2">Hello!!</message>
</messages>
</app>
```
```json
[
{
"id": "deploy",
"execute-command": "/root/my-server/deployment.sh",
"command-working-directory": "/root/my-server",
"trigger-rule": {
"and": [
{
"match": {
"type": "value",
"parameter": {
"source": "payload",
"name": "app.users.user.0.-name"
},
"value": "Jeff"
}
},
{
"match": {
"type": "value",
"parameter": {
"source": "payload",
"name": "app.messages.message.#text"
},
"value": "Hello!!"
}
},
],
}
}
]
```
## Multipart Form Data
Example of a [Plex Media Server webhook](https://support.plex.tv/articles/115002267687-webhooks/).
The Plex Media Server will send two parts: payload and thumb.
We only care about the payload part.
```json
[
{
"id": "plex",
"execute-command": "play-command.sh",
"parse-parameters-as-json": [
{
"source": "payload",
"name": "payload"
}
],
"trigger-rule":
{
"match":
{
"type": "value",
"parameter": {
"source": "payload",
"name": "payload.event"
},
"value": "media.play"
}
}
}
]
```
Each part of a multipart form data body will have a `Content-Disposition` header.
Some example headers:
```
Content-Disposition: form-data; name="payload"
Content-Disposition: form-data; name="thumb"; filename="thumb.jpg"
```
We key off of the `name` attribute in the `Content-Disposition` value.
## Pass string arguments to command
To pass simple string arguments to a command, use the `string` parameter source.
The following example will pass two static string parameters ("-e 123123") to the
`execute-command` before appending the `pusher.email` value from the payload:
```json
[
{
"id": "webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"pass-arguments-to-command":
[
{
"source": "string",
"name": "-e"
},
{
"source": "string",
"name": "123123"
},
{
"source": "payload",
"name": "pusher.email"
}
]
}
]
```
+286
View File
@@ -0,0 +1,286 @@
# Hook rules
### Table of Contents
* [And](#and)
* [Or](#or)
* [Not](#not)
* [Multi-level](#multi-level)
* [Match](#match)
* [Match value](#match-value)
* [Match regex](#match-regex)
* [Match payload-hmac-sha1](#match-payload-hmac-sha1)
* [Match payload-hmac-sha256](#match-payload-hmac-sha256)
* [Match payload-hmac-sha512](#match-payload-hmac-sha512)
* [Match Whitelisted IP range](#match-whitelisted-ip-range)
* [Match scalr-signature](#match-scalr-signature)
## And
*And rule* will evaluate to _true_, if and only if all of the sub rules evaluate to _true_.
```json
{
"and":
[
{
"match":
{
"type": "value",
"value": "refs/heads/master",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
},
{
"match":
{
"type": "regex",
"regex": ".*",
"parameter":
{
"source": "payload",
"name": "repository.owner.name"
}
}
}
]
}
```
## Or
*Or rule* will evaluate to _true_, if any of the sub rules evaluate to _true_.
```json
{
"or":
[
{
"match":
{
"type": "value",
"value": "refs/heads/master",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
},
{
"match":
{
"type": "value",
"value": "refs/heads/development",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
]
}
```
## Not
*Not rule* will evaluate to _true_, if and only if the sub rule evaluates to _false_.
```json
{
"not":
{
"match":
{
"type": "value",
"value": "refs/heads/development",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
}
```
## Multi-level
```json
{
"and": [
{
"match": {
"parameter": {
"source": "header",
"name": "X-Hub-Signature"
},
"type": "payload-hmac-sha1",
"secret": "mysecret"
}
},
{
"or": [
{
"match":
{
"parameter":
{
"source": "payload",
"name": "ref"
},
"type": "value",
"value": "refs/heads/master"
}
},
{
"match":
{
"parameter":
{
"source": "header",
"name": "X-GitHub-Event"
},
"type": "value",
"value": "ping"
}
}
]
}
]
}
```
## Match
*Match rule* will evaluate to _true_, if and only if the referenced value in the `parameter` field satisfies the `type`-specific rule.
*Please note:* Due to technical reasons, _number_ and _boolean_ values in the _match rule_ must be wrapped around with a pair of quotes.
### Match value
```json
{
"match":
{
"type": "value",
"value": "refs/heads/development",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
```
### Match regex
For the regex syntax, check out <http://golang.org/pkg/regexp/syntax/>
```json
{
"match":
{
"type": "regex",
"regex": ".*",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
```
### Match payload-hmac-sha1
Validate the HMAC of the payload using the SHA1 hash and the given *secret*.
```json
{
"match":
{
"type": "payload-hmac-sha1",
"secret": "yoursecret",
"parameter":
{
"source": "header",
"name": "X-Hub-Signature"
}
}
}
```
Note that if multiple signatures were passed via a comma separated string, each
will be tried unless a match is found. For example:
```
X-Hub-Signature: sha1=the-first-signature,sha1=the-second-signature
```
### Match payload-hmac-sha256
Validate the HMAC of the payload using the SHA256 hash and the given *secret*.
```json
{
"match":
{
"type": "payload-hmac-sha256",
"secret": "yoursecret",
"parameter":
{
"source": "header",
"name": "X-Signature"
}
}
}
```
Note that if multiple signatures were passed via a comma separated string, each
will be tried unless a match is found. For example:
```
X-Hub-Signature: sha256=the-first-signature,sha256=the-second-signature
```
### Match payload-hmac-sha512
Validate the HMAC of the payload using the SHA512 hash and the given *secret*.
```json
{
"match":
{
"type": "payload-hmac-sha512",
"secret": "yoursecret",
"parameter":
{
"source": "header",
"name": "X-Signature"
}
}
}
```
Note that if multiple signatures were passed via a comma separated string, each
will be tried unless a match is found. For example:
```
X-Hub-Signature: sha512=the-first-signature,sha512=the-second-signature
```
### Match Whitelisted IP range
The IP can be IPv4- or IPv6-formatted, using [CIDR notation](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#CIDR_blocks). To match a single IP address only, use `/32`.
```json
{
"match":
{
"type": "ip-whitelist",
"ip-range": "192.168.0.1/24"
}
}
```
### Match scalr-signature
The trigger rule checks the scalr signature and also checks that the request was signed less than 5 minutes before it was received.
A unqiue signing key is generated for each webhook endpoint URL you register in Scalr.
Given the time check make sure that NTP is enabled on both your Scalr and webhook server to prevent any issues
```json
{
"match":
{
"type": "scalr-signature",
"secret": "Scalr-provided signing key"
}
}
```
+134
View File
@@ -0,0 +1,134 @@
# Referencing request values
There are four types of request values:
1. HTTP Request Header values
```json
{
"source": "header",
"name": "Header-Name"
}
```
2. HTTP Query parameters
```json
{
"source": "url",
"name": "parameter-name"
}
```
3. HTTP Request parameters
```json
{
"source": "request",
"name": "method"
}
```
```json
{
"source": "request",
"name": "remote-addr"
}
```
4. Payload (JSON or form-value encoded)
```json
{
"source": "payload",
"name": "parameter-name"
}
```
*Note:* For JSON encoded payload, you can reference nested values using the dot-notation.
For example, if you have following JSON payload
```json
{
"commits": [
{
"commit": {
"id": 1
}
}, {
"commit": {
"id": 2
}
}
]
}
```
You can reference the first commit id as
```json
{
"source": "payload",
"name": "commits.0.commit.id"
}
```
If the payload contains a key with the specified name "commits.0.commit.id", then the value of that key has priority over the dot-notation referencing.
4. XML Payload
Referencing XML payload parameters is much like the JSON examples above, but XML is more complex.
Element attributes are prefixed by a hyphen (`-`).
Element values are prefixed by a pound (`#`).
Take the following XML payload:
```xml
<app>
<users>
<user id="1" name="Jeff" />
<user id="2" name="Sally" />
</users>
<messages>
<message id="1" from_user="1" to_user="2">Hello!!</message>
</messages>
</app>
```
To access a given `user` element, you must treat them as an array.
So `app.users.user.0.name` yields `Jeff`.
Since there's only one `message` tag, it's not treated as an array.
So `app.messages.message.id` yields `1`.
To access the text within the `message` tag, you would use: `app.messages.message.#text`.
If you are referencing values for environment, you can use `envname` property to set the name of the environment variable like so
```json
{
"source": "url",
"name": "q",
"envname": "QUERY"
}
```
to get the QUERY environment variable set to the `q` parameter passed in the query string.
# Special cases
If you want to pass the entire payload as JSON string to your command you can use
```json
{
"source": "entire-payload"
}
```
for headers you can use
```json
{
"source": "entire-headers"
}
```
and for query variables you can use
```json
{
"source": "entire-query"
}
```
+76
View File
@@ -0,0 +1,76 @@
# Templates in Webhook
[`webhook`][w] can parse a hooks configuration file as a Go template when given the `-template` [CLI parameter](Webhook-Parameters.md).
In additional to the [built-in Go template functions and features][tt], `webhook` provides a `getenv` template function for inserting environment variables into a templated configuration file.
## Example Usage
In the example JSON template file below (YAML is also supported), the `payload-hmac-sha1` matching rule looks up the HMAC secret from the environment using the `getenv` template function.
Additionally, the result is piped through the built-in Go template function `js` to ensure that the result is a well-formed Javascript/JSON string.
```
{
"id": "webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"response-message": "I got the payload!",
"response-headers":
[
{
"name": "Access-Control-Allow-Origin",
"value": "*"
}
],
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "head_commit.id"
},
{
"source": "payload",
"name": "pusher.name"
},
{
"source": "payload",
"name": "pusher.email"
}
],
"trigger-rule":
{
"and":
[
{
"match":
{
"type": "payload-hmac-sha1",
"secret": "{{ getenv "XXXTEST_SECRET" | js }}",
"parameter":
{
"source": "header",
"name": "X-Hub-Signature"
}
}
},
{
"match":
{
"type": "value",
"value": "refs/heads/master",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
]
}
}
]
```
[w]: https://github.com/adnanh/webhook
[tt]: https://golang.org/pkg/text/template/
+62
View File
@@ -0,0 +1,62 @@
# Webhook parameters
```
Usage of webhook:
-cert string
path to the HTTPS certificate pem file (default "cert.pem")
-cipher-suites string
comma-separated list of supported TLS cipher suites
-debug
show debug output
-header value
response header to return, specified in format name=value, use multiple times to set multiple headers
-hooks value
path to the json file containing defined hooks the webhook should serve, use multiple times to load from different files
-hotreload
watch hooks file for changes and reload them automatically
-http-methods string
globally restrict allowed HTTP methods; separate methods with comma
-ip string
ip the webhook should serve hooks on (default "0.0.0.0")
-key string
path to the HTTPS certificate private key pem file (default "key.pem")
-list-cipher-suites
list available TLS cipher suites
-logfile string
send log output to a file; implicitly enables verbose logging
-nopanic
do not panic if hooks cannot be loaded when webhook is not running in verbose mode
-pidfile string
create PID file at the given path
-port int
port the webhook should serve hooks on (default 9000)
-secure
use HTTPS instead of HTTP
-setgid int
set group ID after opening listening port; must be used with setuid
-setuid int
set user ID after opening listening port; must be used with setgid
-template
parse hooks file as a Go template
-tls-min-version string
minimum TLS version (1.0, 1.1, 1.2, 1.3) (default "1.2")
-urlprefix string
url prefix to use for served hooks (protocol://yourserver:port/PREFIX/:hook-id) (default "hooks")
-verbose
show verbose output
-version
display webhook version and quit
-x-request-id
use X-Request-Id header, if present, as request ID
-x-request-id-limit int
truncate X-Request-Id header to limit; default no limit
```
Use any of the above specified flags to override their default behavior.
# Live reloading hooks
If you are running an OS that supports the HUP or USR1 signal, you can use it to trigger hooks reload from hooks file, without restarting the webhook instance.
```bash
kill -USR1 webhookpid
kill -HUP webhookpid
```
Binary file not shown.

After

Width:  |  Height:  |  Size: 912 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 KiB

+12
View File
@@ -0,0 +1,12 @@
// +build linux windows
package main
import (
"errors"
"runtime"
)
func dropPrivileges(uid, gid int) error {
return errors.New("setuid and setgid not supported on " + runtime.GOOS)
}
+21
View File
@@ -0,0 +1,21 @@
// +build !windows,!linux
package main
import (
"syscall"
)
func dropPrivileges(uid, gid int) error {
err := syscall.Setgid(gid)
if err != nil {
return err
}
err = syscall.Setuid(uid)
if err != nil {
return err
}
return nil
}
+19
View File
@@ -0,0 +1,19 @@
module github.com/adnanh/webhook
go 1.14
require (
github.com/antonmedv/expr v1.8.9
github.com/clbanning/mxj v1.8.4
github.com/dustin/go-humanize v1.0.0
github.com/fsnotify/fsnotify v1.4.7 // indirect
github.com/ghodss/yaml v1.0.0
github.com/go-chi/chi v4.0.2+incompatible
github.com/gofrs/uuid v3.2.0+incompatible
github.com/google/cel-go v0.7.0 // indirect
github.com/gorilla/mux v1.7.3
github.com/kr/pretty v0.1.0 // indirect
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect
gopkg.in/fsnotify.v1 v1.4.2
)
+142
View File
@@ -0,0 +1,142 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/DATA-DOG/go-sqlmock v1.3.3/go.mod h1:f/Ixk793poVmq4qj/V1dPUg2JEAKC73Q5eFN3EC/SaM=
github.com/antlr/antlr4 v0.0.0-20200503195918-621b933c7a7f h1:0cEys61Sr2hUBEXfNV8eyQP01oZuBgoMeHunebPirK8=
github.com/antlr/antlr4 v0.0.0-20200503195918-621b933c7a7f/go.mod h1:T7PbCXFs94rrTttyxjbyT5+/1V8T2TYDejxUfHJjw1Y=
github.com/antonmedv/expr v1.8.9 h1:O9stiHmHHww9b4ozhPx7T6BK7fXfOCHJ8ybxf0833zw=
github.com/antonmedv/expr v1.8.9/go.mod h1:5qsM3oLGDND7sDmQGDXHkYfkjYMUX14qsgqmHhwGEk8=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/clbanning/mxj v1.8.4 h1:HuhwZtbyvyOw+3Z1AowPkU87JkJUSv751ELWaiTpj8I=
github.com/clbanning/mxj v1.8.4/go.mod h1:BVjHeAH+rl9rs6f+QIpeRl0tfu10SXn1pUSa5PVGJng=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/davecgh/go-spew v0.0.0-20161028175848-04cdfd42973b/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/gdamore/encoding v1.0.0/go.mod h1:alR0ol34c49FCSBLjhosxzcPHQbf2trDkoo5dl+VrEg=
github.com/gdamore/tcell v1.3.0/go.mod h1:Hjvr+Ofd+gLglo7RYKxxnzCBmev3BzsS67MebKS4zMM=
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/go-chi/chi v4.0.2+incompatible h1:maB6vn6FqCxrpz4FqWdh4+lwpyZIQS7YEAUcHlgXVRs=
github.com/go-chi/chi v4.0.2+incompatible/go.mod h1:eB3wogJHnLi3x/kFX2A+IbTBlXxmMeXJVKy9tTv1XzQ=
github.com/gofrs/uuid v3.2.0+incompatible h1:y12jRkkFxsd7GpqdSZ+/KCs/fJbqpEXSGd4+jfEaewE=
github.com/gofrs/uuid v3.2.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.3 h1:JjCZWpVbqXDqFVmTfYWEVTMIYrL/NPdPSCHPJ0T/raM=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/google/cel-go v0.7.0 h1:J0J8RSCJW+SdB53YPwPSm2m1Kfz1tqGXdwMuIAVRU9o=
github.com/google/cel-go v0.7.0/go.mod h1:4EtyFAHT5xNr0Msu0MJjyGxPUgdr9DlcaPyzLt/kkt8=
github.com/google/cel-spec v0.5.0/go.mod h1:Nwjgxy5CbjlPrtCWjeDjUyKMl8w41YBYGjsyDdqk0xA=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/mux v1.7.3 h1:gnP5JzjVOuiZD07fKKToCAOjS0yOpj/qPETTXCCS6hw=
github.com/gorilla/mux v1.7.3/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/lucasb-eyer/go-colorful v1.0.2/go.mod h1:0MS4r+7BZKSJ5mw4/S5MPN+qHFF1fYclkSPilDOKW0s=
github.com/lucasb-eyer/go-colorful v1.0.3/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/mattn/go-runewidth v0.0.4/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.8/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/pmezard/go-difflib v0.0.0-20151028094244-d8ed2627bdf0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/rivo/tview v0.0.0-20200219210816-cd38d7432498/go.mod h1:6lkG1x+13OShEf0EaOCaTQYyB7d5nSbb181KtjlS+84=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/sanity-io/litter v1.2.0/go.mod h1:JF6pZUFgu2Q0sBZ+HSV35P8TVPI1TTzEwyu9FXAw2W4=
github.com/stoewer/go-strcase v1.2.0 h1:Z2iHWqGXH00XYgqDmNgQbIBxf3wrNq0F3feEy0ainaU=
github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v0.0.0-20161117074351-18a02ba4a312/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553 h1:efeOvDhwQ29Dj3SdAV/MJf8oukgn+8D8WgaCaRMchF8=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190626150813-e07cf5db2756/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8 h1:JA8d3MPx/IToSyXZG/RhwYEtfrKO1Fxrqe8KrkiLXKM=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527 h1:uYVVQ9WP/Ds2ROhcaGPeIdVq0RIXVLwsHlnvJ+cT1So=
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20201102152239-715cce707fb0 h1:d0rYPqjQfVuFe+tZgv4PHt2hNxK79MRXX7PaD/A5ynA=
google.golang.org/genproto v0.0.0-20201102152239-715cce707fb0/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
google.golang.org/protobuf v1.25.0 h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/fsnotify.v1 v1.4.2 h1:AwZiD/bIUttYJ+n/k1UwlSUsM+VSE6id7UAnSKqQ+Tc=
gopkg.in/fsnotify.v1 v1.4.2/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7 h1:+t9dhfO+GNOIGJof6kPOAenx7YgrZMTdRPV+EsnPabk=
gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
-550
View File
@@ -1,550 +0,0 @@
package hook
import (
"crypto/hmac"
"crypto/sha1"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"net/textproto"
"reflect"
"regexp"
"strconv"
"strings"
)
// Constants used to specify the parameter source
const (
SourceHeader string = "header"
SourceQuery string = "url"
SourceQueryAlias string = "query"
SourcePayload string = "payload"
SourceString string = "string"
SourceEntirePayload string = "entire-payload"
SourceEntireQuery string = "entire-query"
SourceEntireHeaders string = "entire-headers"
)
const (
// EnvNamespace is the prefix used for passing arguments into the command
// environment.
EnvNamespace string = "HOOK_"
)
// SignatureError describes an invalid payload signature passed to Hook.
type SignatureError struct {
Signature string
}
func (e *SignatureError) Error() string {
if e == nil {
return "<nil>"
}
return fmt.Sprintf("invalid payload signature %s", e.Signature)
}
// ArgumentError describes an invalid argument passed to Hook.
type ArgumentError struct {
Argument Argument
}
func (e *ArgumentError) Error() string {
if e == nil {
return "<nil>"
}
return fmt.Sprintf("couldn't retrieve argument for %+v", e.Argument)
}
// SourceError describes an invalid source passed to Hook.
type SourceError struct {
Argument Argument
}
func (e *SourceError) Error() string {
if e == nil {
return "<nil>"
}
return fmt.Sprintf("invalid source for argument %+v", e.Argument)
}
// ParseError describes an error parsing user input.
type ParseError struct {
Err error
}
func (e *ParseError) Error() string {
if e == nil {
return "<nil>"
}
return e.Err.Error()
}
// CheckPayloadSignature calculates and verifies SHA1 signature of the given payload
func CheckPayloadSignature(payload []byte, secret string, signature string) (string, error) {
if strings.HasPrefix(signature, "sha1=") {
signature = signature[5:]
}
mac := hmac.New(sha1.New, []byte(secret))
_, err := mac.Write(payload)
if err != nil {
return "", err
}
expectedMAC := hex.EncodeToString(mac.Sum(nil))
if !hmac.Equal([]byte(signature), []byte(expectedMAC)) {
return expectedMAC, &SignatureError{signature}
}
return expectedMAC, err
}
// ReplaceParameter replaces parameter value with the passed value in the passed map
// (please note you should pass pointer to the map, because we're modifying it)
// based on the passed string
func ReplaceParameter(s string, params interface{}, value interface{}) bool {
if params == nil {
return false
}
if paramsValue := reflect.ValueOf(params); paramsValue.Kind() == reflect.Slice {
if paramsValueSliceLength := paramsValue.Len(); paramsValueSliceLength > 0 {
if p := strings.SplitN(s, ".", 2); len(p) > 1 {
index, err := strconv.ParseUint(p[0], 10, 64)
if err != nil || paramsValueSliceLength <= int(index) {
return false
}
return ReplaceParameter(p[1], params.([]interface{})[index], value)
}
}
return false
}
if p := strings.SplitN(s, ".", 2); len(p) > 1 {
if pValue, ok := params.(map[string]interface{})[p[0]]; ok {
return ReplaceParameter(p[1], pValue, value)
}
} else {
if _, ok := (*params.(*map[string]interface{}))[p[0]]; ok {
(*params.(*map[string]interface{}))[p[0]] = value
return true
}
}
return false
}
// GetParameter extracts interface{} value based on the passed string
func GetParameter(s string, params interface{}) (interface{}, bool) {
if params == nil {
return nil, false
}
if paramsValue := reflect.ValueOf(params); paramsValue.Kind() == reflect.Slice {
if paramsValueSliceLength := paramsValue.Len(); paramsValueSliceLength > 0 {
if p := strings.SplitN(s, ".", 2); len(p) > 1 {
index, err := strconv.ParseUint(p[0], 10, 64)
if err != nil || paramsValueSliceLength <= int(index) {
return nil, false
}
return GetParameter(p[1], params.([]interface{})[index])
}
index, err := strconv.ParseUint(s, 10, 64)
if err != nil || paramsValueSliceLength <= int(index) {
return nil, false
}
return params.([]interface{})[index], true
}
return nil, false
}
if p := strings.SplitN(s, ".", 2); len(p) > 1 {
if paramsValue := reflect.ValueOf(params); paramsValue.Kind() == reflect.Map {
if pValue, ok := params.(map[string]interface{})[p[0]]; ok {
return GetParameter(p[1], pValue)
}
} else {
return nil, false
}
} else {
if pValue, ok := params.(map[string]interface{})[p[0]]; ok {
return pValue, true
}
}
return nil, false
}
// ExtractParameterAsString extracts value from interface{} as string based on the passed string
func ExtractParameterAsString(s string, params interface{}) (string, bool) {
if pValue, ok := GetParameter(s, params); ok {
return fmt.Sprintf("%v", pValue), true
}
return "", false
}
// Argument type specifies the parameter key name and the source it should
// be extracted from
type Argument struct {
Source string `json:"source,omitempty"`
Name string `json:"name,omitempty"`
EnvName string `json:"envname,omitempty"`
}
// Get Argument method returns the value for the Argument's key name
// based on the Argument's source
func (ha *Argument) Get(headers, query, payload *map[string]interface{}) (string, bool) {
var source *map[string]interface{}
key := ha.Name
switch ha.Source {
case SourceHeader:
source = headers
key = textproto.CanonicalMIMEHeaderKey(ha.Name)
case SourceQuery, SourceQueryAlias:
source = query
case SourcePayload:
source = payload
case SourceString:
return ha.Name, true
case SourceEntirePayload:
r, err := json.Marshal(payload)
if err != nil {
return "", false
}
return string(r), true
case SourceEntireHeaders:
r, err := json.Marshal(headers)
if err != nil {
return "", false
}
return string(r), true
case SourceEntireQuery:
r, err := json.Marshal(query)
if err != nil {
return "", false
}
return string(r), true
}
if source != nil {
return ExtractParameterAsString(key, *source)
}
return "", false
}
// Header is a structure containing header name and it's value
type Header struct {
Name string `json:"name"`
Value string `json:"value"`
}
// ResponseHeaders is a slice of Header objects
type ResponseHeaders []Header
func (h *ResponseHeaders) String() string {
// a 'hack' to display name=value in flag usage listing
if len(*h) == 0 {
return "name=value"
}
result := make([]string, len(*h))
for idx, responseHeader := range *h {
result[idx] = fmt.Sprintf("%s=%s", responseHeader.Name, responseHeader.Value)
}
return fmt.Sprint(strings.Join(result, ", "))
}
// Set method appends new Header object from header=value notation
func (h *ResponseHeaders) Set(value string) error {
splitResult := strings.SplitN(value, "=", 2)
if len(splitResult) != 2 {
return errors.New("header flag must be in name=value format")
}
*h = append(*h, Header{Name: splitResult[0], Value: splitResult[1]})
return nil
}
// Hook type is a structure containing details for a single hook
type Hook struct {
ID string `json:"id,omitempty"`
ExecuteCommand string `json:"execute-command,omitempty"`
CommandWorkingDirectory string `json:"command-working-directory,omitempty"`
ResponseMessage string `json:"response-message,omitempty"`
ResponseHeaders ResponseHeaders `json:"response-headers,omitempty"`
CaptureCommandOutput bool `json:"include-command-output-in-response,omitempty"`
PassEnvironmentToCommand []Argument `json:"pass-environment-to-command,omitempty"`
PassArgumentsToCommand []Argument `json:"pass-arguments-to-command,omitempty"`
JSONStringParameters []Argument `json:"parse-parameters-as-json,omitempty"`
TriggerRule *Rules `json:"trigger-rule,omitempty"`
TriggerRuleMismatchHttpResponseCode int `json:"trigger-rule-mismatch-http-response-code,omitempty"`
}
// ParseJSONParameters decodes specified arguments to JSON objects and replaces the
// string with the newly created object
func (h *Hook) ParseJSONParameters(headers, query, payload *map[string]interface{}) []error {
var errors = make([]error, 0)
for i := range h.JSONStringParameters {
if arg, ok := h.JSONStringParameters[i].Get(headers, query, payload); ok {
var newArg map[string]interface{}
decoder := json.NewDecoder(strings.NewReader(string(arg)))
decoder.UseNumber()
err := decoder.Decode(&newArg)
if err != nil {
errors = append(errors, &ParseError{err})
continue
}
var source *map[string]interface{}
switch h.JSONStringParameters[i].Source {
case SourceHeader:
source = headers
case SourcePayload:
source = payload
case SourceQuery, SourceQueryAlias:
source = query
}
if source != nil {
key := h.JSONStringParameters[i].Name
if h.JSONStringParameters[i].Source == SourceHeader {
key = textproto.CanonicalMIMEHeaderKey(h.JSONStringParameters[i].Name)
}
ReplaceParameter(key, source, newArg)
} else {
errors = append(errors, &SourceError{h.JSONStringParameters[i]})
}
} else {
errors = append(errors, &ArgumentError{h.JSONStringParameters[i]})
}
}
if len(errors) > 0 {
return errors
}
return nil
}
// ExtractCommandArguments creates a list of arguments, based on the
// PassArgumentsToCommand property that is ready to be used with exec.Command()
func (h *Hook) ExtractCommandArguments(headers, query, payload *map[string]interface{}) ([]string, []error) {
var args = make([]string, 0)
var errors = make([]error, 0)
args = append(args, h.ExecuteCommand)
for i := range h.PassArgumentsToCommand {
if arg, ok := h.PassArgumentsToCommand[i].Get(headers, query, payload); ok {
args = append(args, arg)
} else {
args = append(args, "")
errors = append(errors, &ArgumentError{h.PassArgumentsToCommand[i]})
}
}
if len(errors) > 0 {
return args, errors
}
return args, nil
}
// ExtractCommandArgumentsForEnv creates a list of arguments in key=value
// format, based on the PassEnvironmentToCommand property that is ready to be used
// with exec.Command().
func (h *Hook) ExtractCommandArgumentsForEnv(headers, query, payload *map[string]interface{}) ([]string, []error) {
var args = make([]string, 0)
var errors = make([]error, 0)
for i := range h.PassEnvironmentToCommand {
if arg, ok := h.PassEnvironmentToCommand[i].Get(headers, query, payload); ok {
if h.PassEnvironmentToCommand[i].EnvName != "" {
// first try to use the EnvName if specified
args = append(args, h.PassEnvironmentToCommand[i].EnvName+"="+arg)
} else {
// then fallback on the name
args = append(args, EnvNamespace+h.PassEnvironmentToCommand[i].Name+"="+arg)
}
} else {
errors = append(errors, &ArgumentError{h.PassEnvironmentToCommand[i]})
}
}
if len(errors) > 0 {
return args, errors
}
return args, nil
}
// Hooks is an array of Hook objects
type Hooks []Hook
// LoadFromFile attempts to load hooks from specified JSON file
func (h *Hooks) LoadFromFile(path string) error {
if path == "" {
return nil
}
// parse hook file for hooks
file, e := ioutil.ReadFile(path)
if e != nil {
return e
}
e = json.Unmarshal(file, h)
return e
}
// Match iterates through Hooks and returns first one that matches the given ID,
// if no hook matches the given ID, nil is returned
func (h *Hooks) Match(id string) *Hook {
for i := range *h {
if (*h)[i].ID == id {
return &(*h)[i]
}
}
return nil
}
// Rules is a structure that contains one of the valid rule types
type Rules struct {
And *AndRule `json:"and,omitempty"`
Or *OrRule `json:"or,omitempty"`
Not *NotRule `json:"not,omitempty"`
Match *MatchRule `json:"match,omitempty"`
}
// Evaluate finds the first rule property that is not nil and returns the value
// it evaluates to
func (r Rules) Evaluate(headers, query, payload *map[string]interface{}, body *[]byte) (bool, error) {
switch {
case r.And != nil:
return r.And.Evaluate(headers, query, payload, body)
case r.Or != nil:
return r.Or.Evaluate(headers, query, payload, body)
case r.Not != nil:
return r.Not.Evaluate(headers, query, payload, body)
case r.Match != nil:
return r.Match.Evaluate(headers, query, payload, body)
}
return false, nil
}
// AndRule will evaluate to true if and only if all of the ChildRules evaluate to true
type AndRule []Rules
// Evaluate AndRule will return true if and only if all of ChildRules evaluate to true
func (r AndRule) Evaluate(headers, query, payload *map[string]interface{}, body *[]byte) (bool, error) {
res := true
for _, v := range r {
rv, err := v.Evaluate(headers, query, payload, body)
if err != nil {
return false, err
}
res = res && rv
if res == false {
return res, nil
}
}
return res, nil
}
// OrRule will evaluate to true if any of the ChildRules evaluate to true
type OrRule []Rules
// Evaluate OrRule will return true if any of ChildRules evaluate to true
func (r OrRule) Evaluate(headers, query, payload *map[string]interface{}, body *[]byte) (bool, error) {
res := false
for _, v := range r {
rv, err := v.Evaluate(headers, query, payload, body)
if err != nil {
return false, err
}
res = res || rv
if res == true {
return res, nil
}
}
return res, nil
}
// NotRule will evaluate to true if any and only if the ChildRule evaluates to false
type NotRule Rules
// Evaluate NotRule will return true if and only if ChildRule evaluates to false
func (r NotRule) Evaluate(headers, query, payload *map[string]interface{}, body *[]byte) (bool, error) {
rv, err := Rules(r).Evaluate(headers, query, payload, body)
return !rv, err
}
// MatchRule will evaluate to true based on the type
type MatchRule struct {
Type string `json:"type,omitempty"`
Regex string `json:"regex,omitempty"`
Secret string `json:"secret,omitempty"`
Value string `json:"value,omitempty"`
Parameter Argument `json:"parameter,omitempty"`
}
// Constants for the MatchRule type
const (
MatchValue string = "value"
MatchRegex string = "regex"
MatchHashSHA1 string = "payload-hash-sha1"
)
// Evaluate MatchRule will return based on the type
func (r MatchRule) Evaluate(headers, query, payload *map[string]interface{}, body *[]byte) (bool, error) {
if arg, ok := r.Parameter.Get(headers, query, payload); ok {
switch r.Type {
case MatchValue:
return arg == r.Value, nil
case MatchRegex:
return regexp.MatchString(r.Regex, arg)
case MatchHashSHA1:
_, err := CheckPayloadSignature(*body, r.Secret, arg)
return err == nil, err
}
}
return false, nil
}
-418
View File
@@ -1,418 +0,0 @@
package hook
import (
"reflect"
"strings"
"testing"
)
var checkPayloadSignatureTests = []struct {
payload []byte
secret string
signature string
mac string
ok bool
}{
{[]byte(`{"a": "z"}`), "secret", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", true},
{[]byte(`{"a": "z"}`), "secret", "sha1=b17e04cbb22afa8ffbff8796fc1894ed27badd9e", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", true},
// failures
{[]byte(`{"a": "z"}`), "secret", "XXXe04cbb22afa8ffbff8796fc1894ed27badd9e", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", false},
{[]byte(`{"a": "z"}`), "secreX", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", "900225703e9342328db7307692736e2f7cc7b36e", false},
}
func TestCheckPayloadSignature(t *testing.T) {
for _, tt := range checkPayloadSignatureTests {
mac, err := CheckPayloadSignature(tt.payload, tt.secret, tt.signature)
if (err == nil) != tt.ok || mac != tt.mac {
t.Errorf("failed to check payload signature {%q, %q, %q}:\nexpected {mac:%#v, ok:%#v},\ngot {mac:%#v, ok:%#v}", tt.payload, tt.secret, tt.signature, tt.mac, tt.ok, mac, (err == nil))
}
if err != nil && strings.Contains(err.Error(), tt.mac) {
t.Errorf("error message should not disclose expected mac: %s", err)
}
}
}
var extractParameterTests = []struct {
s string
params interface{}
value string
ok bool
}{
{"a", map[string]interface{}{"a": "z"}, "z", true},
{"a.b", map[string]interface{}{"a": map[string]interface{}{"b": "z"}}, "z", true},
{"a.b.c", map[string]interface{}{"a": map[string]interface{}{"b": map[string]interface{}{"c": "z"}}}, "z", true},
{"a.b.0", map[string]interface{}{"a": map[string]interface{}{"b": []interface{}{"x", "y", "z"}}}, "x", true},
{"a.1.b", map[string]interface{}{"a": []interface{}{map[string]interface{}{"b": "y"}, map[string]interface{}{"b": "z"}}}, "z", true},
{"a.1.b.c", map[string]interface{}{"a": []interface{}{map[string]interface{}{"b": map[string]interface{}{"c": "y"}}, map[string]interface{}{"b": map[string]interface{}{"c": "z"}}}}, "z", true},
// failures
{"check_nil", nil, "", false},
{"a.X", map[string]interface{}{"a": map[string]interface{}{"b": "z"}}, "", false}, // non-existent parameter reference
{"a.X.c", map[string]interface{}{"a": []interface{}{map[string]interface{}{"b": "y"}, map[string]interface{}{"b": "z"}}}, "", false}, // non-integer slice index
{"a.-1.b", map[string]interface{}{"a": []interface{}{map[string]interface{}{"b": "y"}, map[string]interface{}{"b": "z"}}}, "", false}, // negative slice index
{"a.500.b", map[string]interface{}{"a": map[string]interface{}{"b": "z"}}, "", false}, // non-existent slice
{"a.501.b", map[string]interface{}{"a": []interface{}{map[string]interface{}{"b": "y"}, map[string]interface{}{"b": "z"}}}, "", false}, // non-existent slice index
{"a.502.b", map[string]interface{}{"a": []interface{}{}}, "", false}, // non-existent slice index
{"a.b.503", map[string]interface{}{"a": map[string]interface{}{"b": []interface{}{"x", "y", "z"}}}, "", false}, // trailing, non-existent slice index
{"a.b", interface{}("a"), "", false}, // non-map, non-slice input
}
func TestExtractParameter(t *testing.T) {
for _, tt := range extractParameterTests {
value, ok := ExtractParameterAsString(tt.s, tt.params)
if ok != tt.ok || value != tt.value {
t.Errorf("failed to extract parameter %q:\nexpected {value:%#v, ok:%#v},\ngot {value:%#v, ok:%#v}", tt.s, tt.value, tt.ok, value, ok)
}
}
}
var argumentGetTests = []struct {
source, name string
headers, query, payload *map[string]interface{}
value string
ok bool
}{
{"header", "a", &map[string]interface{}{"A": "z"}, nil, nil, "z", true},
{"url", "a", nil, &map[string]interface{}{"a": "z"}, nil, "z", true},
{"payload", "a", nil, nil, &map[string]interface{}{"a": "z"}, "z", true},
{"string", "a", nil, nil, &map[string]interface{}{"a": "z"}, "a", true},
// failures
{"header", "a", nil, &map[string]interface{}{"a": "z"}, &map[string]interface{}{"a": "z"}, "", false}, // nil headers
{"url", "a", &map[string]interface{}{"A": "z"}, nil, &map[string]interface{}{"a": "z"}, "", false}, // nil query
{"payload", "a", &map[string]interface{}{"A": "z"}, &map[string]interface{}{"a": "z"}, nil, "", false}, // nil payload
{"foo", "a", &map[string]interface{}{"A": "z"}, nil, nil, "", false}, // invalid source
}
func TestArgumentGet(t *testing.T) {
for _, tt := range argumentGetTests {
a := Argument{tt.source, tt.name, ""}
value, ok := a.Get(tt.headers, tt.query, tt.payload)
if ok != tt.ok || value != tt.value {
t.Errorf("failed to get {%q, %q}:\nexpected {value:%#v, ok:%#v},\ngot {value:%#v, ok:%#v}", tt.source, tt.name, tt.value, tt.ok, value, ok)
}
}
}
var hookParseJSONParametersTests = []struct {
params []Argument
headers, query, payload *map[string]interface{}
rheaders, rquery, rpayload *map[string]interface{}
ok bool
}{
{[]Argument{Argument{"header", "a", ""}}, &map[string]interface{}{"A": `{"b": "y"}`}, nil, nil, &map[string]interface{}{"A": map[string]interface{}{"b": "y"}}, nil, nil, true},
{[]Argument{Argument{"url", "a", ""}}, nil, &map[string]interface{}{"a": `{"b": "y"}`}, nil, nil, &map[string]interface{}{"a": map[string]interface{}{"b": "y"}}, nil, true},
{[]Argument{Argument{"payload", "a", ""}}, nil, nil, &map[string]interface{}{"a": `{"b": "y"}`}, nil, nil, &map[string]interface{}{"a": map[string]interface{}{"b": "y"}}, true},
{[]Argument{Argument{"header", "z", ""}}, &map[string]interface{}{"Z": `{}`}, nil, nil, &map[string]interface{}{"Z": map[string]interface{}{}}, nil, nil, true},
// failures
{[]Argument{Argument{"header", "z", ""}}, &map[string]interface{}{"Z": ``}, nil, nil, &map[string]interface{}{"Z": ``}, nil, nil, false}, // empty string
{[]Argument{Argument{"header", "y", ""}}, &map[string]interface{}{"X": `{}`}, nil, nil, &map[string]interface{}{"X": `{}`}, nil, nil, false}, // missing parameter
{[]Argument{Argument{"string", "z", ""}}, &map[string]interface{}{"Z": ``}, nil, nil, &map[string]interface{}{"Z": ``}, nil, nil, false}, // invalid argument source
}
func TestHookParseJSONParameters(t *testing.T) {
for _, tt := range hookParseJSONParametersTests {
h := &Hook{JSONStringParameters: tt.params}
err := h.ParseJSONParameters(tt.headers, tt.query, tt.payload)
if (err == nil) != tt.ok || !reflect.DeepEqual(tt.headers, tt.rheaders) {
t.Errorf("failed to parse %v:\nexpected %#v, ok: %v\ngot %#v, ok: %v", tt.params, *tt.rheaders, tt.ok, *tt.headers, (err == nil))
}
}
}
var hookExtractCommandArgumentsTests = []struct {
exec string
args []Argument
headers, query, payload *map[string]interface{}
value []string
ok bool
}{
{"test", []Argument{Argument{"header", "a", ""}}, &map[string]interface{}{"A": "z"}, nil, nil, []string{"test", "z"}, true},
// failures
{"fail", []Argument{Argument{"payload", "a", ""}}, &map[string]interface{}{"A": "z"}, nil, nil, []string{"fail", ""}, false},
}
func TestHookExtractCommandArguments(t *testing.T) {
for _, tt := range hookExtractCommandArgumentsTests {
h := &Hook{ExecuteCommand: tt.exec, PassArgumentsToCommand: tt.args}
value, err := h.ExtractCommandArguments(tt.headers, tt.query, tt.payload)
if (err == nil) != tt.ok || !reflect.DeepEqual(value, tt.value) {
t.Errorf("failed to extract args {cmd=%q, args=%v}:\nexpected %#v, ok: %v\ngot %#v, ok: %v", tt.exec, tt.args, tt.value, tt.ok, value, (err == nil))
}
}
}
// Here we test the extraction of env variables when the user defined a hook
// with the "pass-environment-to-command" directive
// we test both cases where the name of the data is used as the name of the
// env key & the case where the hook definition sets the env var name to a
// fixed value using the envname construct like so::
// [
// {
// "id": "push",
// "execute-command": "bb2mm",
// "command-working-directory": "/tmp",
// "pass-environment-to-command":
// [
// {
// "source": "entire-payload",
// "envname": "PAYLOAD"
// },
// ]
// }
// ]
var hookExtractCommandArgumentsForEnvTests = []struct {
exec string
args []Argument
headers, query, payload *map[string]interface{}
value []string
ok bool
}{
// successes
{
"test",
[]Argument{Argument{"header", "a", ""}},
&map[string]interface{}{"A": "z"}, nil, nil,
[]string{"HOOK_a=z"},
true,
},
{
"test",
[]Argument{Argument{"header", "a", "MYKEY"}},
&map[string]interface{}{"A": "z"}, nil, nil,
[]string{"MYKEY=z"},
true,
},
// failures
{
"fail",
[]Argument{Argument{"payload", "a", ""}},
&map[string]interface{}{"A": "z"}, nil, nil,
[]string{},
false,
},
}
func TestHookExtractCommandArgumentsForEnv(t *testing.T) {
for _, tt := range hookExtractCommandArgumentsForEnvTests {
h := &Hook{ExecuteCommand: tt.exec, PassEnvironmentToCommand: tt.args}
value, err := h.ExtractCommandArgumentsForEnv(tt.headers, tt.query, tt.payload)
if (err == nil) != tt.ok || !reflect.DeepEqual(value, tt.value) {
t.Errorf("failed to extract args for env {cmd=%q, args=%v}:\nexpected %#v, ok: %v\ngot %#v, ok: %v", tt.exec, tt.args, tt.value, tt.ok, value, (err == nil))
}
}
}
var hooksLoadFromFileTests = []struct {
path string
ok bool
}{
{"../hooks.json.example", true},
{"", true},
// failures
{"missing.json", false},
}
func TestHooksLoadFromFile(t *testing.T) {
for _, tt := range hooksLoadFromFileTests {
h := &Hooks{}
err := h.LoadFromFile(tt.path)
if (err == nil) != tt.ok {
t.Errorf(err.Error())
}
}
}
var hooksMatchTests = []struct {
id string
hooks Hooks
value *Hook
}{
{"a", Hooks{Hook{ID: "a"}}, &Hook{ID: "a"}},
{"X", Hooks{Hook{ID: "a"}}, new(Hook)},
}
func TestHooksMatch(t *testing.T) {
for _, tt := range hooksMatchTests {
value := tt.hooks.Match(tt.id)
if reflect.DeepEqual(reflect.ValueOf(value), reflect.ValueOf(tt.value)) {
t.Errorf("failed to match %q:\nexpected %#v,\ngot %#v", tt.id, tt.value, value)
}
}
}
var matchRuleTests = []struct {
typ, regex, secret, value string
param Argument
headers, query, payload *map[string]interface{}
body []byte
ok bool
err bool
}{
{"value", "", "", "z", Argument{"header", "a", ""}, &map[string]interface{}{"A": "z"}, nil, nil, []byte{}, true, false},
{"regex", "^z", "", "z", Argument{"header", "a", ""}, &map[string]interface{}{"A": "z"}, nil, nil, []byte{}, true, false},
{"payload-hash-sha1", "", "secret", "", Argument{"header", "a", ""}, &map[string]interface{}{"A": "b17e04cbb22afa8ffbff8796fc1894ed27badd9e"}, nil, nil, []byte(`{"a": "z"}`), true, false},
// failures
{"value", "", "", "X", Argument{"header", "a", ""}, &map[string]interface{}{"A": "z"}, nil, nil, []byte{}, false, false},
{"regex", "^X", "", "", Argument{"header", "a", ""}, &map[string]interface{}{"A": "z"}, nil, nil, []byte{}, false, false},
{"value", "", "2", "X", Argument{"header", "a", ""}, &map[string]interface{}{"Y": "z"}, nil, nil, []byte{}, false, false}, // reference invalid header
// errors
{"regex", "*", "", "", Argument{"header", "a", ""}, &map[string]interface{}{"A": "z"}, nil, nil, []byte{}, false, true}, // invalid regex
{"payload-hash-sha1", "", "secret", "", Argument{"header", "a", ""}, &map[string]interface{}{"A": ""}, nil, nil, []byte{}, false, true}, // invalid hmac
}
func TestMatchRule(t *testing.T) {
for i, tt := range matchRuleTests {
r := MatchRule{tt.typ, tt.regex, tt.secret, tt.value, tt.param}
ok, err := r.Evaluate(tt.headers, tt.query, tt.payload, &tt.body)
if ok != tt.ok || (err != nil) != tt.err {
t.Errorf("%d failed to match %#v:\nexpected ok: %#v, err: %v\ngot ok: %#v, err: %v", i, r, tt.ok, tt.err, ok, (err != nil))
}
}
}
var andRuleTests = []struct {
desc string // description of the test case
rule AndRule
headers, query, payload *map[string]interface{}
body []byte
ok bool
err bool
}{
{
"(a=z, b=y): a=z && b=y",
AndRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", ""}}},
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", ""}}},
},
&map[string]interface{}{"A": "z", "B": "y"}, nil, nil, []byte{},
true, false,
},
{
"(a=z, b=Y): a=z && b=y",
AndRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", ""}}},
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", ""}}},
},
&map[string]interface{}{"A": "z", "B": "Y"}, nil, nil, []byte{},
false, false,
},
// Complex test to cover Rules.Evaluate
{
"(a=z, b=y, c=x, d=w=, e=X, f=X): a=z && (b=y && c=x) && (d=w || e=v) && !f=u",
AndRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", ""}}},
{
And: &AndRule{
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", ""}}},
{Match: &MatchRule{"value", "", "", "x", Argument{"header", "c", ""}}},
},
},
{
Or: &OrRule{
{Match: &MatchRule{"value", "", "", "w", Argument{"header", "d", ""}}},
{Match: &MatchRule{"value", "", "", "v", Argument{"header", "e", ""}}},
},
},
{
Not: &NotRule{
Match: &MatchRule{"value", "", "", "u", Argument{"header", "f", ""}},
},
},
},
&map[string]interface{}{"A": "z", "B": "y", "C": "x", "D": "w", "E": "X", "F": "X"}, nil, nil, []byte{},
true, false,
},
{"empty rule", AndRule{{}}, nil, nil, nil, nil, false, false},
// failures
{
"invalid rule",
AndRule{{Match: &MatchRule{"value", "", "", "X", Argument{"header", "a", ""}}}},
&map[string]interface{}{"Y": "z"}, nil, nil, nil,
false, false,
},
}
func TestAndRule(t *testing.T) {
for _, tt := range andRuleTests {
ok, err := tt.rule.Evaluate(tt.headers, tt.query, tt.payload, &tt.body)
if ok != tt.ok || (err != nil) != tt.err {
t.Errorf("failed to match %#v:\nexpected ok: %#v, err: %v\ngot ok: %#v, err: %v", tt.desc, tt.ok, tt.err, ok, err)
}
}
}
var orRuleTests = []struct {
desc string // description of the test case
rule OrRule
headers, query, payload *map[string]interface{}
body []byte
ok bool
err bool
}{
{
"(a=z, b=X): a=z || b=y",
OrRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", ""}}},
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", ""}}},
},
&map[string]interface{}{"A": "z", "B": "X"}, nil, nil, []byte{},
true, false,
},
{
"(a=X, b=y): a=z || b=y",
OrRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", ""}}},
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", ""}}},
},
&map[string]interface{}{"A": "X", "B": "y"}, nil, nil, []byte{},
true, false,
},
{
"(a=Z, b=Y): a=z || b=y",
OrRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", ""}}},
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", ""}}},
},
&map[string]interface{}{"A": "Z", "B": "Y"}, nil, nil, []byte{},
false, false,
},
// failures
{
"invalid rule",
OrRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", ""}}},
},
&map[string]interface{}{"Y": "Z"}, nil, nil, []byte{},
false, false,
},
}
func TestOrRule(t *testing.T) {
for _, tt := range orRuleTests {
ok, err := tt.rule.Evaluate(tt.headers, tt.query, tt.payload, &tt.body)
if ok != tt.ok || (err != nil) != tt.err {
t.Errorf("%#v:\nexpected ok: %#v, err: %v\ngot ok: %#v err: %v", tt.desc, tt.ok, tt.err, ok, err)
}
}
}
var notRuleTests = []struct {
desc string // description of the test case
rule NotRule
headers, query, payload *map[string]interface{}
body []byte
ok bool
err bool
}{
{"(a=z): !a=X", NotRule{Match: &MatchRule{"value", "", "", "X", Argument{"header", "a", ""}}}, &map[string]interface{}{"A": "z"}, nil, nil, []byte{}, true, false},
{"(a=z): !a=z", NotRule{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", ""}}}, &map[string]interface{}{"A": "z"}, nil, nil, []byte{}, false, false},
}
func TestNotRule(t *testing.T) {
for _, tt := range notRuleTests {
ok, err := tt.rule.Evaluate(tt.headers, tt.query, tt.payload, &tt.body)
if ok != tt.ok || (err != nil) != tt.err {
t.Errorf("failed to match %#v:\nexpected ok: %#v, err: %v\ngot ok: %#v, err: %v", tt.rule, tt.ok, tt.err, ok, err)
}
}
}
+55
View File
@@ -0,0 +1,55 @@
[
{
"id": "webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"response-message": "I got the payload!",
"response-headers":
[
{
"name": "Access-Control-Allow-Origin",
"value": "*"
}
],
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "head_commit.id"
},
{
"source": "payload",
"name": "pusher.name"
},
{
"source": "payload",
"name": "pusher.email"
}
],
"trigger-rule":
{
"or":
[
{
"match":
{
"type": "expression",
"expression": "\"ada\" in Payload"
}
},
{
"match":
{
"type": "value",
"value": "refs/heads/master",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
]
}
}
]
+1 -1
View File
@@ -33,7 +33,7 @@
{
"match":
{
"type": "payload-hash-sha1",
"type": "payload-hmac-sha1",
"secret": "mysecret",
"parameter":
{
+60
View File
@@ -0,0 +1,60 @@
[
{
"id": "webhook",
"execute-command": "/home/adnan/redeploy-go-webhook.sh",
"command-working-directory": "/home/adnan/go",
"response-message": "I got the payload!",
"response-headers":
[
{
"name": "Access-Control-Allow-Origin",
"value": "*"
}
],
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "head_commit.id"
},
{
"source": "payload",
"name": "pusher.name"
},
{
"source": "payload",
"name": "pusher.email"
}
],
"trigger-rule":
{
"and":
[
{
"match":
{
"type": "payload-hmac-sha1",
"secret": "{{ getenv "XXXTEST_SECRET" | js }}",
"parameter":
{
"source": "header",
"name": "X-Hub-Signature"
}
}
},
{
"match":
{
"type": "value",
"value": "refs/heads/master",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
]
}
}
]
+28
View File
@@ -0,0 +1,28 @@
- id: webhook
execute-command: /home/adnan/redeploy-go-webhook.sh
command-working-directory: /home/adnan/go
response-message: I got the payload!
response-headers:
- name: Access-Control-Allow-Origin
value: '*'
pass-arguments-to-command:
- source: payload
name: head_commit.id
- source: payload
name: pusher.name
- source: payload
name: pusher.email
trigger-rule:
and:
- match:
type: payload-hmac-sha1
secret: mysecret
parameter:
source: header
name: X-Hub-Signature
- match:
type: value
value: refs/heads/master
parameter:
source: payload
name: ref
+28
View File
@@ -0,0 +1,28 @@
- id: webhook
execute-command: /home/adnan/redeploy-go-webhook.sh
command-working-directory: /home/adnan/go
response-message: I got the payload!
response-headers:
- name: Access-Control-Allow-Origin
value: '*'
pass-arguments-to-command:
- source: payload
name: head_commit.id
- source: payload
name: pusher.name
- source: payload
name: pusher.email
trigger-rule:
and:
- match:
type: payload-hmac-sha1
secret: "{{ getenv "XXXTEST_SECRET" | js }}"
parameter:
source: header
name: X-Hub-Signature
- match:
type: value
value: refs/heads/master
parameter:
source: payload
name: ref
+987
View File
@@ -0,0 +1,987 @@
package hook
import (
"bytes"
"crypto/hmac"
"crypto/sha1"
"crypto/sha256"
"crypto/sha512"
"crypto/subtle"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"hash"
"io/ioutil"
"log"
"math"
"net"
"net/textproto"
"os"
"reflect"
"regexp"
"strconv"
"strings"
"text/template"
"time"
"github.com/antonmedv/expr"
"github.com/ghodss/yaml"
)
// Constants used to specify the parameter source
const (
SourceHeader string = "header"
SourceQuery string = "url"
SourceQueryAlias string = "query"
SourcePayload string = "payload"
SourceRawRequestBody string = "raw-request-body"
SourceRequest string = "request"
SourceString string = "string"
SourceEntirePayload string = "entire-payload"
SourceEntireQuery string = "entire-query"
SourceEntireHeaders string = "entire-headers"
)
const (
// EnvNamespace is the prefix used for passing arguments into the command
// environment.
EnvNamespace string = "HOOK_"
)
// ParameterNodeError describes an error walking a parameter node.
type ParameterNodeError struct {
key string
}
func (e *ParameterNodeError) Error() string {
if e == nil {
return "<nil>"
}
return fmt.Sprintf("parameter node not found: %s", e.key)
}
// IsParameterNodeError returns whether err is of type ParameterNodeError.
func IsParameterNodeError(err error) bool {
switch err.(type) {
case *ParameterNodeError:
return true
default:
return false
}
}
// SignatureError describes an invalid payload signature passed to Hook.
type SignatureError struct {
Signature string
Signatures []string
emptyPayload bool
}
func (e *SignatureError) Error() string {
if e == nil {
return "<nil>"
}
var empty string
if e.emptyPayload {
empty = " on empty payload"
}
if e.Signatures != nil {
return fmt.Sprintf("invalid payload signatures %s%s", e.Signatures, empty)
}
return fmt.Sprintf("invalid payload signature %s%s", e.Signature, empty)
}
// IsSignatureError returns whether err is of type SignatureError.
func IsSignatureError(err error) bool {
switch err.(type) {
case *SignatureError:
return true
default:
return false
}
}
// ArgumentError describes an invalid argument passed to Hook.
type ArgumentError struct {
Argument Argument
}
func (e *ArgumentError) Error() string {
if e == nil {
return "<nil>"
}
return fmt.Sprintf("couldn't retrieve argument for %+v", e.Argument)
}
// SourceError describes an invalid source passed to Hook.
type SourceError struct {
Argument Argument
}
func (e *SourceError) Error() string {
if e == nil {
return "<nil>"
}
return fmt.Sprintf("invalid source for argument %+v", e.Argument)
}
// ParseError describes an error parsing user input.
type ParseError struct {
Err error
}
func (e *ParseError) Error() string {
if e == nil {
return "<nil>"
}
return e.Err.Error()
}
// ExtractCommaSeparatedValues will extract the values matching the key.
func ExtractCommaSeparatedValues(source, prefix string) []string {
parts := strings.Split(source, ",")
values := make([]string, 0)
for _, part := range parts {
if strings.HasPrefix(part, prefix) {
values = append(values, strings.TrimPrefix(part, prefix))
}
}
return values
}
// ExtractSignatures will extract all the signatures from the source.
func ExtractSignatures(source, prefix string) []string {
// If there are multiple possible matches, let the comma seperated extractor
// do it's work.
if strings.Contains(source, ",") {
return ExtractCommaSeparatedValues(source, prefix)
}
// There were no commas, so just trim the prefix (if it even exists) and
// pass it back.
return []string{
strings.TrimPrefix(source, prefix),
}
}
// ValidateMAC will verify that the expected mac for the given hash will match
// the one provided.
func ValidateMAC(payload []byte, mac hash.Hash, signatures []string) (string, error) {
// Write the payload to the provided hash.
_, err := mac.Write(payload)
if err != nil {
return "", err
}
actualMAC := hex.EncodeToString(mac.Sum(nil))
for _, signature := range signatures {
if hmac.Equal([]byte(signature), []byte(actualMAC)) {
return actualMAC, err
}
}
e := &SignatureError{Signatures: signatures}
if len(payload) == 0 {
e.emptyPayload = true
}
return actualMAC, e
}
// CheckPayloadSignature calculates and verifies SHA1 signature of the given payload
func CheckPayloadSignature(payload []byte, secret, signature string) (string, error) {
if secret == "" {
return "", errors.New("signature validation secret can not be empty")
}
// Extract the signatures.
signatures := ExtractSignatures(signature, "sha1=")
// Validate the MAC.
return ValidateMAC(payload, hmac.New(sha1.New, []byte(secret)), signatures)
}
// CheckPayloadSignature256 calculates and verifies SHA256 signature of the given payload
func CheckPayloadSignature256(payload []byte, secret, signature string) (string, error) {
if secret == "" {
return "", errors.New("signature validation secret can not be empty")
}
// Extract the signatures.
signatures := ExtractSignatures(signature, "sha256=")
// Validate the MAC.
return ValidateMAC(payload, hmac.New(sha256.New, []byte(secret)), signatures)
}
// CheckPayloadSignature512 calculates and verifies SHA512 signature of the given payload
func CheckPayloadSignature512(payload []byte, secret, signature string) (string, error) {
if secret == "" {
return "", errors.New("signature validation secret can not be empty")
}
// Extract the signatures.
signatures := ExtractSignatures(signature, "sha512=")
// Validate the MAC.
return ValidateMAC(payload, hmac.New(sha512.New, []byte(secret)), signatures)
}
func CheckScalrSignature(r *Request, signingKey string, checkDate bool) (bool, error) {
if r.Headers == nil {
return false, nil
}
// Check for the signature and date headers
if _, ok := r.Headers["X-Signature"]; !ok {
return false, nil
}
if _, ok := r.Headers["Date"]; !ok {
return false, nil
}
if signingKey == "" {
return false, errors.New("signature validation signing key can not be empty")
}
providedSignature := r.Headers["X-Signature"].(string)
dateHeader := r.Headers["Date"].(string)
mac := hmac.New(sha1.New, []byte(signingKey))
mac.Write(r.Body)
mac.Write([]byte(dateHeader))
expectedSignature := hex.EncodeToString(mac.Sum(nil))
if !hmac.Equal([]byte(providedSignature), []byte(expectedSignature)) {
return false, &SignatureError{Signature: providedSignature}
}
if !checkDate {
return true, nil
}
// Example format: Fri 08 Sep 2017 11:24:32 UTC
date, err := time.Parse("Mon 02 Jan 2006 15:04:05 MST", dateHeader)
if err != nil {
return false, err
}
now := time.Now()
delta := math.Abs(now.Sub(date).Seconds())
if delta > 300 {
return false, &SignatureError{Signature: "outdated"}
}
return true, nil
}
// CheckIPWhitelist makes sure the provided remote address (of the form IP:port) falls within the provided IP range
// (in CIDR form or a single IP address).
func CheckIPWhitelist(remoteAddr, ipRange string) (bool, error) {
// Extract IP address from remote address.
// IPv6 addresses will likely be surrounded by [].
ip := strings.Trim(remoteAddr, " []")
if i := strings.LastIndex(ip, ":"); i != -1 {
ip = ip[:i]
ip = strings.Trim(ip, " []")
}
parsedIP := net.ParseIP(ip)
if parsedIP == nil {
return false, fmt.Errorf("invalid IP address found in remote address '%s'", remoteAddr)
}
for _, r := range strings.Fields(ipRange) {
// Extract IP range in CIDR form. If a single IP address is provided, turn it into CIDR form.
if !strings.Contains(r, "/") {
r = r + "/32"
}
_, cidr, err := net.ParseCIDR(r)
if err != nil {
return false, err
}
if cidr.Contains(parsedIP) {
return true, nil
}
}
return false, nil
}
type ExpressionEnv struct {
Headers, Query, Payload map[string]interface{}
}
func CheckExpression(request *Request, expression string) (bool, error) {
program, err := expr.Compile(expression, expr.Env(ExpressionEnv{}))
if err != nil {
return false, err
}
out, err := expr.Run(program, ExpressionEnv{
Headers: request.Headers,
Query: request.Query,
Payload: request.Payload,
})
if err != nil {
return false, err
}
return out == true, nil
}
// ReplaceParameter replaces parameter value with the passed value in the passed map
// (please note you should pass pointer to the map, because we're modifying it)
// based on the passed string
func ReplaceParameter(s string, params, value interface{}) bool {
if params == nil {
return false
}
if paramsValue := reflect.ValueOf(params); paramsValue.Kind() == reflect.Slice {
if paramsValueSliceLength := paramsValue.Len(); paramsValueSliceLength > 0 {
if p := strings.SplitN(s, ".", 2); len(p) > 1 {
index, err := strconv.ParseUint(p[0], 10, 64)
if err != nil || paramsValueSliceLength <= int(index) {
return false
}
return ReplaceParameter(p[1], params.([]interface{})[index], value)
}
}
return false
}
if p := strings.SplitN(s, ".", 2); len(p) > 1 {
if pValue, ok := params.(map[string]interface{})[p[0]]; ok {
return ReplaceParameter(p[1], pValue, value)
}
} else {
if _, ok := (*params.(*map[string]interface{}))[p[0]]; ok {
(*params.(*map[string]interface{}))[p[0]] = value
return true
}
}
return false
}
// GetParameter extracts interface{} value based on the passed string
func GetParameter(s string, params interface{}) (interface{}, error) {
if params == nil {
return nil, errors.New("no parameters")
}
paramsValue := reflect.ValueOf(params)
switch paramsValue.Kind() {
case reflect.Slice:
paramsValueSliceLength := paramsValue.Len()
if paramsValueSliceLength > 0 {
if p := strings.SplitN(s, ".", 2); len(p) > 1 {
index, err := strconv.ParseUint(p[0], 10, 64)
if err != nil || paramsValueSliceLength <= int(index) {
return nil, &ParameterNodeError{s}
}
return GetParameter(p[1], params.([]interface{})[index])
}
index, err := strconv.ParseUint(s, 10, 64)
if err != nil || paramsValueSliceLength <= int(index) {
return nil, &ParameterNodeError{s}
}
return params.([]interface{})[index], nil
}
return nil, &ParameterNodeError{s}
case reflect.Map:
// Check for raw key
if v, ok := params.(map[string]interface{})[s]; ok {
return v, nil
}
// Checked for dotted references
p := strings.SplitN(s, ".", 2)
if pValue, ok := params.(map[string]interface{})[p[0]]; ok {
if len(p) > 1 {
return GetParameter(p[1], pValue)
}
return pValue, nil
}
}
return nil, &ParameterNodeError{s}
}
// ExtractParameterAsString extracts value from interface{} as string based on
// the passed string. Complex data types are rendered as JSON instead of the Go
// Stringer format.
func ExtractParameterAsString(s string, params interface{}) (string, error) {
pValue, err := GetParameter(s, params)
if err != nil {
return "", err
}
switch v := reflect.ValueOf(pValue); v.Kind() {
case reflect.Array, reflect.Map, reflect.Slice:
r, err := json.Marshal(pValue)
if err != nil {
return "", err
}
return string(r), nil
default:
return fmt.Sprintf("%v", pValue), nil
}
}
// Argument type specifies the parameter key name and the source it should
// be extracted from
type Argument struct {
Source string `json:"source,omitempty"`
Name string `json:"name,omitempty"`
EnvName string `json:"envname,omitempty"`
Base64Decode bool `json:"base64decode,omitempty"`
}
// Get Argument method returns the value for the Argument's key name
// based on the Argument's source
func (ha *Argument) Get(r *Request) (string, error) {
var source *map[string]interface{}
key := ha.Name
switch ha.Source {
case SourceHeader:
source = &r.Headers
key = textproto.CanonicalMIMEHeaderKey(ha.Name)
case SourceQuery, SourceQueryAlias:
source = &r.Query
case SourcePayload:
source = &r.Payload
case SourceString:
return ha.Name, nil
case SourceRawRequestBody:
return string(r.Body), nil
case SourceRequest:
if r == nil || r.RawRequest == nil {
return "", errors.New("request is nil")
}
switch strings.ToLower(ha.Name) {
case "remote-addr":
return r.RawRequest.RemoteAddr, nil
case "method":
return r.RawRequest.Method, nil
default:
return "", fmt.Errorf("unsupported request key: %q", ha.Name)
}
case SourceEntirePayload:
res, err := json.Marshal(&r.Payload)
if err != nil {
return "", err
}
return string(res), nil
case SourceEntireHeaders:
res, err := json.Marshal(&r.Headers)
if err != nil {
return "", err
}
return string(res), nil
case SourceEntireQuery:
res, err := json.Marshal(&r.Query)
if err != nil {
return "", err
}
return string(res), nil
}
if source != nil {
return ExtractParameterAsString(key, *source)
}
return "", errors.New("no source for value retrieval")
}
// Header is a structure containing header name and it's value
type Header struct {
Name string `json:"name"`
Value string `json:"value"`
}
// ResponseHeaders is a slice of Header objects
type ResponseHeaders []Header
func (h *ResponseHeaders) String() string {
// a 'hack' to display name=value in flag usage listing
if len(*h) == 0 {
return "name=value"
}
result := make([]string, len(*h))
for idx, responseHeader := range *h {
result[idx] = fmt.Sprintf("%s=%s", responseHeader.Name, responseHeader.Value)
}
return strings.Join(result, ", ")
}
// Set method appends new Header object from header=value notation
func (h *ResponseHeaders) Set(value string) error {
splitResult := strings.SplitN(value, "=", 2)
if len(splitResult) != 2 {
return errors.New("header flag must be in name=value format")
}
*h = append(*h, Header{Name: splitResult[0], Value: splitResult[1]})
return nil
}
// HooksFiles is a slice of String
type HooksFiles []string
func (h *HooksFiles) String() string {
if len(*h) == 0 {
return "hooks.json"
}
return strings.Join(*h, ", ")
}
// Set method appends new string
func (h *HooksFiles) Set(value string) error {
*h = append(*h, value)
return nil
}
// Hook type is a structure containing details for a single hook
type Hook struct {
ID string `json:"id,omitempty"`
ExecuteCommand string `json:"execute-command,omitempty"`
CommandWorkingDirectory string `json:"command-working-directory,omitempty"`
ResponseMessage string `json:"response-message,omitempty"`
ResponseHeaders ResponseHeaders `json:"response-headers,omitempty"`
CaptureCommandOutput bool `json:"include-command-output-in-response,omitempty"`
CaptureCommandOutputOnError bool `json:"include-command-output-in-response-on-error,omitempty"`
PassEnvironmentToCommand []Argument `json:"pass-environment-to-command,omitempty"`
PassArgumentsToCommand []Argument `json:"pass-arguments-to-command,omitempty"`
PassFileToCommand []Argument `json:"pass-file-to-command,omitempty"`
JSONStringParameters []Argument `json:"parse-parameters-as-json,omitempty"`
TriggerRule *Rules `json:"trigger-rule,omitempty"`
TriggerRuleMismatchHttpResponseCode int `json:"trigger-rule-mismatch-http-response-code,omitempty"`
TriggerSignatureSoftFailures bool `json:"trigger-signature-soft-failures,omitempty"`
IncomingPayloadContentType string `json:"incoming-payload-content-type,omitempty"`
SuccessHttpResponseCode int `json:"success-http-response-code,omitempty"`
HTTPMethods []string `json:"http-methods"`
}
// ParseJSONParameters decodes specified arguments to JSON objects and replaces the
// string with the newly created object
func (h *Hook) ParseJSONParameters(r *Request) []error {
errors := make([]error, 0)
for i := range h.JSONStringParameters {
arg, err := h.JSONStringParameters[i].Get(r)
if err != nil {
errors = append(errors, &ArgumentError{h.JSONStringParameters[i]})
} else {
var newArg map[string]interface{}
decoder := json.NewDecoder(strings.NewReader(string(arg)))
decoder.UseNumber()
err := decoder.Decode(&newArg)
if err != nil {
errors = append(errors, &ParseError{err})
continue
}
var source *map[string]interface{}
switch h.JSONStringParameters[i].Source {
case SourceHeader:
source = &r.Headers
case SourcePayload:
source = &r.Payload
case SourceQuery, SourceQueryAlias:
source = &r.Query
}
if source != nil {
key := h.JSONStringParameters[i].Name
if h.JSONStringParameters[i].Source == SourceHeader {
key = textproto.CanonicalMIMEHeaderKey(h.JSONStringParameters[i].Name)
}
ReplaceParameter(key, source, newArg)
} else {
errors = append(errors, &SourceError{h.JSONStringParameters[i]})
}
}
}
if len(errors) > 0 {
return errors
}
return nil
}
// ExtractCommandArguments creates a list of arguments, based on the
// PassArgumentsToCommand property that is ready to be used with exec.Command()
func (h *Hook) ExtractCommandArguments(r *Request) ([]string, []error) {
args := make([]string, 0)
errors := make([]error, 0)
args = append(args, h.ExecuteCommand)
for i := range h.PassArgumentsToCommand {
arg, err := h.PassArgumentsToCommand[i].Get(r)
if err != nil {
args = append(args, "")
errors = append(errors, &ArgumentError{h.PassArgumentsToCommand[i]})
continue
}
args = append(args, arg)
}
if len(errors) > 0 {
return args, errors
}
return args, nil
}
// ExtractCommandArgumentsForEnv creates a list of arguments in key=value
// format, based on the PassEnvironmentToCommand property that is ready to be used
// with exec.Command().
func (h *Hook) ExtractCommandArgumentsForEnv(r *Request) ([]string, []error) {
args := make([]string, 0)
errors := make([]error, 0)
for i := range h.PassEnvironmentToCommand {
arg, err := h.PassEnvironmentToCommand[i].Get(r)
if err != nil {
errors = append(errors, &ArgumentError{h.PassEnvironmentToCommand[i]})
continue
}
if h.PassEnvironmentToCommand[i].EnvName != "" {
// first try to use the EnvName if specified
args = append(args, h.PassEnvironmentToCommand[i].EnvName+"="+arg)
} else {
// then fallback on the name
args = append(args, EnvNamespace+h.PassEnvironmentToCommand[i].Name+"="+arg)
}
}
if len(errors) > 0 {
return args, errors
}
return args, nil
}
// FileParameter describes a pass-file-to-command instance to be stored as file
type FileParameter struct {
File *os.File
EnvName string
Data []byte
}
// ExtractCommandArgumentsForFile creates a list of arguments in key=value
// format, based on the PassFileToCommand property that is ready to be used
// with exec.Command().
func (h *Hook) ExtractCommandArgumentsForFile(r *Request) ([]FileParameter, []error) {
args := make([]FileParameter, 0)
errors := make([]error, 0)
for i := range h.PassFileToCommand {
arg, err := h.PassFileToCommand[i].Get(r)
if err != nil {
errors = append(errors, &ArgumentError{h.PassFileToCommand[i]})
continue
}
if h.PassFileToCommand[i].EnvName == "" {
// if no environment-variable name is set, fall-back on the name
log.Printf("no ENVVAR name specified, falling back to [%s]", EnvNamespace+strings.ToUpper(h.PassFileToCommand[i].Name))
h.PassFileToCommand[i].EnvName = EnvNamespace + strings.ToUpper(h.PassFileToCommand[i].Name)
}
var fileContent []byte
if h.PassFileToCommand[i].Base64Decode {
dec, err := base64.StdEncoding.DecodeString(arg)
if err != nil {
log.Printf("error decoding string [%s]", err)
}
fileContent = []byte(dec)
} else {
fileContent = []byte(arg)
}
args = append(args, FileParameter{EnvName: h.PassFileToCommand[i].EnvName, Data: fileContent})
}
if len(errors) > 0 {
return args, errors
}
return args, nil
}
// Hooks is an array of Hook objects
type Hooks []Hook
// LoadFromFile attempts to load hooks from the specified file, which
// can be either JSON or YAML. The asTemplate parameter causes the file
// contents to be parsed as a Go text/template prior to unmarshalling.
func (h *Hooks) LoadFromFile(path string, asTemplate bool) error {
if path == "" {
return nil
}
// parse hook file for hooks
file, e := ioutil.ReadFile(path)
if e != nil {
return e
}
if asTemplate {
funcMap := template.FuncMap{"getenv": getenv}
tmpl, err := template.New("hooks").Funcs(funcMap).Parse(string(file))
if err != nil {
return err
}
var buf bytes.Buffer
err = tmpl.Execute(&buf, nil)
if err != nil {
return err
}
file = buf.Bytes()
}
return yaml.Unmarshal(file, h)
}
// Append appends hooks unless the new hooks contain a hook with an ID that already exists
func (h *Hooks) Append(other *Hooks) error {
for _, hook := range *other {
if h.Match(hook.ID) != nil {
return fmt.Errorf("hook with ID %s is already defined", hook.ID)
}
*h = append(*h, hook)
}
return nil
}
// Match iterates through Hooks and returns first one that matches the given ID,
// if no hook matches the given ID, nil is returned
func (h *Hooks) Match(id string) *Hook {
for i := range *h {
if (*h)[i].ID == id {
return &(*h)[i]
}
}
return nil
}
// Rules is a structure that contains one of the valid rule types
type Rules struct {
And *AndRule `json:"and,omitempty"`
Or *OrRule `json:"or,omitempty"`
Not *NotRule `json:"not,omitempty"`
Match *MatchRule `json:"match,omitempty"`
}
// Evaluate finds the first rule property that is not nil and returns the value
// it evaluates to
func (r Rules) Evaluate(req *Request) (bool, error) {
switch {
case r.And != nil:
return r.And.Evaluate(req)
case r.Or != nil:
return r.Or.Evaluate(req)
case r.Not != nil:
return r.Not.Evaluate(req)
case r.Match != nil:
return r.Match.Evaluate(req)
}
return false, nil
}
// AndRule will evaluate to true if and only if all of the ChildRules evaluate to true
type AndRule []Rules
// Evaluate AndRule will return true if and only if all of ChildRules evaluate to true
func (r AndRule) Evaluate(req *Request) (bool, error) {
res := true
for _, v := range r {
rv, err := v.Evaluate(req)
if err != nil {
return false, err
}
res = res && rv
if !res {
return res, nil
}
}
return res, nil
}
// OrRule will evaluate to true if any of the ChildRules evaluate to true
type OrRule []Rules
// Evaluate OrRule will return true if any of ChildRules evaluate to true
func (r OrRule) Evaluate(req *Request) (bool, error) {
res := false
for _, v := range r {
rv, err := v.Evaluate(req)
if err != nil {
if !IsParameterNodeError(err) {
if !req.AllowSignatureErrors || (req.AllowSignatureErrors && !IsSignatureError(err)) {
return false, err
}
}
}
res = res || rv
if res {
return res, nil
}
}
return res, nil
}
// NotRule will evaluate to true if any and only if the ChildRule evaluates to false
type NotRule Rules
// Evaluate NotRule will return true if and only if ChildRule evaluates to false
func (r NotRule) Evaluate(req *Request) (bool, error) {
rv, err := Rules(r).Evaluate(req)
return !rv, err
}
// MatchRule will evaluate to true based on the type
type MatchRule struct {
Type string `json:"type,omitempty"`
Regex string `json:"regex,omitempty"`
Expression string `json:"expression,omitempty"`
Secret string `json:"secret,omitempty"`
Value string `json:"value,omitempty"`
Parameter Argument `json:"parameter,omitempty"`
IPRange string `json:"ip-range,omitempty"`
}
// Constants for the MatchRule type
const (
MatchValue string = "value"
MatchRegex string = "regex"
MatchExpression string = "expression"
MatchHMACSHA1 string = "payload-hmac-sha1"
MatchHMACSHA256 string = "payload-hmac-sha256"
MatchHMACSHA512 string = "payload-hmac-sha512"
MatchHashSHA1 string = "payload-hash-sha1"
MatchHashSHA256 string = "payload-hash-sha256"
MatchHashSHA512 string = "payload-hash-sha512"
IPWhitelist string = "ip-whitelist"
ScalrSignature string = "scalr-signature"
)
// Evaluate MatchRule will return based on the type
func (r MatchRule) Evaluate(req *Request) (bool, error) {
if r.Type == IPWhitelist {
return CheckIPWhitelist(req.RawRequest.RemoteAddr, r.IPRange)
}
if r.Type == ScalrSignature {
return CheckScalrSignature(req, r.Secret, true)
}
if r.Type == MatchExpression {
return CheckExpression(req, r.Expression)
}
arg, err := r.Parameter.Get(req)
if err == nil {
switch r.Type {
case MatchValue:
return compare(arg, r.Value), nil
case MatchRegex:
return regexp.MatchString(r.Regex, arg)
case MatchHashSHA1:
log.Print(`warn: use of deprecated option payload-hash-sha1; use payload-hmac-sha1 instead`)
fallthrough
case MatchHMACSHA1:
_, err := CheckPayloadSignature(req.Body, r.Secret, arg)
return err == nil, err
case MatchHashSHA256:
log.Print(`warn: use of deprecated option payload-hash-sha256: use payload-hmac-sha256 instead`)
fallthrough
case MatchHMACSHA256:
_, err := CheckPayloadSignature256(req.Body, r.Secret, arg)
return err == nil, err
case MatchHashSHA512:
log.Print(`warn: use of deprecated option payload-hash-sha512: use payload-hmac-sha512 instead`)
fallthrough
case MatchHMACSHA512:
_, err := CheckPayloadSignature512(req.Body, r.Secret, arg)
return err == nil, err
}
}
return false, err
}
// compare is a helper function for constant time string comparisons.
func compare(a, b string) bool {
return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1
}
// getenv provides a template function to retrieve OS environment variables.
func getenv(s string) string {
return os.Getenv(s)
}
+728
View File
@@ -0,0 +1,728 @@
package hook
import (
"net/http"
"os"
"reflect"
"strings"
"testing"
)
func TestGetParameter(t *testing.T) {
for _, test := range []struct {
key string
val interface{}
expect interface{}
ok bool
}{
// True
{"a", map[string]interface{}{"a": "1"}, "1", true},
{"a.b", map[string]interface{}{"a.b": "1"}, "1", true},
{"a.c", map[string]interface{}{"a": map[string]interface{}{"c": 2}}, 2, true},
{"a.1", map[string]interface{}{"a": map[string]interface{}{"1": 3}}, 3, true},
{"a.1", map[string]interface{}{"a": []interface{}{"a", "b"}}, "b", true},
{"0", []interface{}{"a", "b"}, "a", true},
// False
{"z", map[string]interface{}{"a": "1"}, nil, false},
{"a.z", map[string]interface{}{"a": map[string]interface{}{"b": 2}}, nil, false},
{"z.b", map[string]interface{}{"a": map[string]interface{}{"z": 2}}, nil, false},
{"a.2", map[string]interface{}{"a": []interface{}{"a", "b"}}, nil, false},
} {
res, err := GetParameter(test.key, test.val)
if (err == nil) != test.ok {
t.Errorf("unexpected result given {%q, %q}: %s\n", test.key, test.val, err)
}
if !reflect.DeepEqual(res, test.expect) {
t.Errorf("failed given {%q, %q}:\nexpected {%#v}\ngot {%#v}\n", test.key, test.val, test.expect, res)
}
}
}
var checkPayloadSignatureTests = []struct {
payload []byte
secret string
signature string
mac string
ok bool
}{
{[]byte(`{"a": "z"}`), "secret", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", true},
{[]byte(`{"a": "z"}`), "secret", "sha1=b17e04cbb22afa8ffbff8796fc1894ed27badd9e", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", true},
{[]byte(`{"a": "z"}`), "secret", "sha1=XXXe04cbb22afa8ffbff8796fc1894ed27badd9e,sha1=b17e04cbb22afa8ffbff8796fc1894ed27badd9e", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", true},
{[]byte(``), "secret", "25af6174a0fcecc4d346680a72b7ce644b9a88e8", "25af6174a0fcecc4d346680a72b7ce644b9a88e8", true},
// failures
{[]byte(`{"a": "z"}`), "secret", "XXXe04cbb22afa8ffbff8796fc1894ed27badd9e", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", false},
{[]byte(`{"a": "z"}`), "secret", "sha1=XXXe04cbb22afa8ffbff8796fc1894ed27badd9e", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", false},
{[]byte(`{"a": "z"}`), "secret", "sha1=XXXe04cbb22afa8ffbff8796fc1894ed27badd9e,sha1=XXXe04cbb22afa8ffbff8796fc1894ed27badd9e", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", false},
{[]byte(`{"a": "z"}`), "secreX", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", "900225703e9342328db7307692736e2f7cc7b36e", false},
{[]byte(`{"a": "z"}`), "", "b17e04cbb22afa8ffbff8796fc1894ed27badd9e", "", false},
{[]byte(``), "secret", "XXXf6174a0fcecc4d346680a72b7ce644b9a88e8", "25af6174a0fcecc4d346680a72b7ce644b9a88e8", false},
}
func TestCheckPayloadSignature(t *testing.T) {
for _, tt := range checkPayloadSignatureTests {
mac, err := CheckPayloadSignature(tt.payload, tt.secret, tt.signature)
if (err == nil) != tt.ok || mac != tt.mac {
t.Errorf("failed to check payload signature {%q, %q, %q}:\nexpected {mac:%#v, ok:%#v},\ngot {mac:%#v, ok:%#v}", tt.payload, tt.secret, tt.signature, tt.mac, tt.ok, mac, (err == nil))
}
if err != nil && tt.mac != "" && strings.Contains(err.Error(), tt.mac) {
t.Errorf("error message should not disclose expected mac: %s", err)
}
}
}
var checkPayloadSignature256Tests = []struct {
payload []byte
secret string
signature string
mac string
ok bool
}{
{[]byte(`{"a": "z"}`), "secret", "f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", "f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", true},
{[]byte(`{"a": "z"}`), "secret", "sha256=f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", "f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", true},
{[]byte(`{"a": "z"}`), "secret", "sha256=XXX7af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89,sha256=f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", "f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", true},
{[]byte(``), "secret", "f9e66e179b6747ae54108f82f8ade8b3c25d76fd30afde6c395822c530196169", "f9e66e179b6747ae54108f82f8ade8b3c25d76fd30afde6c395822c530196169", true},
// failures
{[]byte(`{"a": "z"}`), "secret", "XXX7af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", "f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", false},
{[]byte(`{"a": "z"}`), "secret", "sha256=XXX7af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", "f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", false},
{[]byte(`{"a": "z"}`), "secret", "sha256=XXX7af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89,sha256=XXX7af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", "f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", false},
{[]byte(`{"a": "z"}`), "", "XXX7af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89", "", false},
{[]byte(``), "secret", "XXX66e179b6747ae54108f82f8ade8b3c25d76fd30afde6c395822c530196169", "f9e66e179b6747ae54108f82f8ade8b3c25d76fd30afde6c395822c530196169", false},
}
func TestCheckPayloadSignature256(t *testing.T) {
for _, tt := range checkPayloadSignature256Tests {
mac, err := CheckPayloadSignature256(tt.payload, tt.secret, tt.signature)
if (err == nil) != tt.ok || mac != tt.mac {
t.Errorf("failed to check payload signature {%q, %q, %q}:\nexpected {mac:%#v, ok:%#v},\ngot {mac:%#v, ok:%#v}", tt.payload, tt.secret, tt.signature, tt.mac, tt.ok, mac, (err == nil))
}
if err != nil && tt.mac != "" && strings.Contains(err.Error(), tt.mac) {
t.Errorf("error message should not disclose expected mac: %s", err)
}
}
}
var checkPayloadSignature512Tests = []struct {
payload []byte
secret string
signature string
mac string
ok bool
}{
{[]byte(`{"a": "z"}`), "secret", "4ab17cc8ec668ead8bf498f87f8f32848c04d5ca3c9bcfcd3db9363f0deb44e580b329502a7fdff633d4d8fca301cc5c94a55a2fec458c675fb0ff2655898324", "4ab17cc8ec668ead8bf498f87f8f32848c04d5ca3c9bcfcd3db9363f0deb44e580b329502a7fdff633d4d8fca301cc5c94a55a2fec458c675fb0ff2655898324", true},
{[]byte(`{"a": "z"}`), "secret", "sha512=4ab17cc8ec668ead8bf498f87f8f32848c04d5ca3c9bcfcd3db9363f0deb44e580b329502a7fdff633d4d8fca301cc5c94a55a2fec458c675fb0ff2655898324", "4ab17cc8ec668ead8bf498f87f8f32848c04d5ca3c9bcfcd3db9363f0deb44e580b329502a7fdff633d4d8fca301cc5c94a55a2fec458c675fb0ff2655898324", true},
{[]byte(``), "secret", "b0e9650c5faf9cd8ae02276671545424104589b3656731ec193b25d01b07561c27637c2d4d68389d6cf5007a8632c26ec89ba80a01c77a6cdd389ec28db43901", "b0e9650c5faf9cd8ae02276671545424104589b3656731ec193b25d01b07561c27637c2d4d68389d6cf5007a8632c26ec89ba80a01c77a6cdd389ec28db43901", true},
// failures
{[]byte(`{"a": "z"}`), "secret", "74a0081f5b5988f4f3e8b8dd34dadc6291611f2e6260635a7e1535f8e95edb97ff520ba8b152e8ca5760ac42639854f3242e29efc81be73a8bf52d474d31ffea", "4ab17cc8ec668ead8bf498f87f8f32848c04d5ca3c9bcfcd3db9363f0deb44e580b329502a7fdff633d4d8fca301cc5c94a55a2fec458c675fb0ff2655898324", false},
{[]byte(`{"a": "z"}`), "", "74a0081f5b5988f4f3e8b8dd34dadc6291611f2e6260635a7e1535f8e95edb97ff520ba8b152e8ca5760ac42639854f3242e29efc81be73a8bf52d474d31ffea", "", false},
{[]byte(``), "secret", "XXX9650c5faf9cd8ae02276671545424104589b3656731ec193b25d01b07561c27637c2d4d68389d6cf5007a8632c26ec89ba80a01c77a6cdd389ec28db43901", "b0e9650c5faf9cd8ae02276671545424104589b3656731ec193b25d01b07561c27637c2d4d68389d6cf5007a8632c26ec89ba80a01c77a6cdd389ec28db43901", false},
}
func TestCheckPayloadSignature512(t *testing.T) {
for _, tt := range checkPayloadSignature512Tests {
mac, err := CheckPayloadSignature512(tt.payload, tt.secret, tt.signature)
if (err == nil) != tt.ok || mac != tt.mac {
t.Errorf("failed to check payload signature {%q, %q, %q}:\nexpected {mac:%#v, ok:%#v},\ngot {mac:%#v, ok:%#v}", tt.payload, tt.secret, tt.signature, tt.mac, tt.ok, mac, (err == nil))
}
if err != nil && tt.mac != "" && strings.Contains(err.Error(), tt.mac) {
t.Errorf("error message should not disclose expected mac: %s", err)
}
}
}
var checkScalrSignatureTests = []struct {
description string
headers map[string]interface{}
body []byte
secret string
expectedSignature string
ok bool
}{
{
"Valid signature",
map[string]interface{}{"Date": "Thu 07 Sep 2017 06:30:04 UTC", "X-Signature": "48e395e38ac48988929167df531eb2da00063a7d"},
[]byte(`{"a": "b"}`), "bilFGi4ZVZUdG+C6r0NIM9tuRq6PaG33R3eBUVhLwMAErGBaazvXe4Gq2DcJs5q+",
"48e395e38ac48988929167df531eb2da00063a7d", true,
},
{
"Wrong signature",
map[string]interface{}{"Date": "Thu 07 Sep 2017 06:30:04 UTC", "X-Signature": "999395e38ac48988929167df531eb2da00063a7d"},
[]byte(`{"a": "b"}`), "bilFGi4ZVZUdG+C6r0NIM9tuRq6PaG33R3eBUVhLwMAErGBaazvXe4Gq2DcJs5q+",
"48e395e38ac48988929167df531eb2da00063a7d", false,
},
{
"Missing Date header",
map[string]interface{}{"X-Signature": "999395e38ac48988929167df531eb2da00063a7d"},
[]byte(`{"a": "b"}`), "bilFGi4ZVZUdG+C6r0NIM9tuRq6PaG33R3eBUVhLwMAErGBaazvXe4Gq2DcJs5q+",
"48e395e38ac48988929167df531eb2da00063a7d", false,
},
{
"Missing X-Signature header",
map[string]interface{}{"Date": "Thu 07 Sep 2017 06:30:04 UTC"},
[]byte(`{"a": "b"}`), "bilFGi4ZVZUdG+C6r0NIM9tuRq6PaG33R3eBUVhLwMAErGBaazvXe4Gq2DcJs5q+",
"48e395e38ac48988929167df531eb2da00063a7d", false,
},
{
"Missing signing key",
map[string]interface{}{"Date": "Thu 07 Sep 2017 06:30:04 UTC", "X-Signature": "48e395e38ac48988929167df531eb2da00063a7d"},
[]byte(`{"a": "b"}`), "",
"48e395e38ac48988929167df531eb2da00063a7d", false,
},
}
func TestCheckScalrSignature(t *testing.T) {
for _, testCase := range checkScalrSignatureTests {
r := &Request{
Headers: testCase.headers,
Body: testCase.body,
}
valid, err := CheckScalrSignature(r, testCase.secret, false)
if valid != testCase.ok {
t.Errorf("failed to check scalr signature fot test case: %s\nexpected ok:%#v, got ok:%#v}",
testCase.description, testCase.ok, valid)
}
if err != nil && testCase.secret != "" && strings.Contains(err.Error(), testCase.expectedSignature) {
t.Errorf("error message should not disclose expected mac: %s on test case %s", err, testCase.description)
}
}
}
var checkIPWhitelistTests = []struct {
addr string
ipRange string
expect bool
ok bool
}{
{"[ 10.0.0.1:1234 ] ", " 10.0.0.1 ", true, true},
{"[ 10.0.0.1:1234 ] ", " 10.0.0.0 ", false, true},
{"[ 10.0.0.1:1234 ] ", " 10.0.0.1 10.0.0.1 ", true, true},
{"[ 10.0.0.1:1234 ] ", " 10.0.0.0/31 ", true, true},
{" [2001:db8:1:2::1:1234] ", " 2001:db8:1::/48 ", true, true},
{" [2001:db8:1:2::1:1234] ", " 2001:db8:1::/48 2001:db8:1::/64", true, true},
{" [2001:db8:1:2::1:1234] ", " 2001:db8:1::/64 ", false, true},
}
func TestCheckIPWhitelist(t *testing.T) {
for _, tt := range checkIPWhitelistTests {
result, err := CheckIPWhitelist(tt.addr, tt.ipRange)
if (err == nil) != tt.ok || result != tt.expect {
t.Errorf("ip whitelist test failed {%q, %q}:\nwant {expect:%#v, ok:%#v},\ngot {result:%#v, ok:%#v}", tt.addr, tt.ipRange, tt.expect, tt.ok, result, err)
}
}
}
var extractParameterTests = []struct {
s string
params interface{}
value string
ok bool
}{
{"a", map[string]interface{}{"a": "z"}, "z", true},
{"a.b", map[string]interface{}{"a": map[string]interface{}{"b": "z"}}, "z", true},
{"a.b.c", map[string]interface{}{"a": map[string]interface{}{"b": map[string]interface{}{"c": "z"}}}, "z", true},
{"a.b.0", map[string]interface{}{"a": map[string]interface{}{"b": []interface{}{"x", "y", "z"}}}, "x", true},
{"a.1.b", map[string]interface{}{"a": []interface{}{map[string]interface{}{"b": "y"}, map[string]interface{}{"b": "z"}}}, "z", true},
{"a.1.b.c", map[string]interface{}{"a": []interface{}{map[string]interface{}{"b": map[string]interface{}{"c": "y"}}, map[string]interface{}{"b": map[string]interface{}{"c": "z"}}}}, "z", true},
{"b", map[string]interface{}{"b": map[string]interface{}{"z": 1}}, `{"z":1}`, true},
{"c", map[string]interface{}{"c": []interface{}{"y", "z"}}, `["y","z"]`, true},
{"d", map[string]interface{}{"d": [2]interface{}{"y", "z"}}, `["y","z"]`, true},
// failures
{"check_nil", nil, "", false},
{"a.X", map[string]interface{}{"a": map[string]interface{}{"b": "z"}}, "", false}, // non-existent parameter reference
{"a.X.c", map[string]interface{}{"a": []interface{}{map[string]interface{}{"b": "y"}, map[string]interface{}{"b": "z"}}}, "", false}, // non-integer slice index
{"a.-1.b", map[string]interface{}{"a": []interface{}{map[string]interface{}{"b": "y"}, map[string]interface{}{"b": "z"}}}, "", false}, // negative slice index
{"a.500.b", map[string]interface{}{"a": map[string]interface{}{"b": "z"}}, "", false}, // non-existent slice
{"a.501.b", map[string]interface{}{"a": []interface{}{map[string]interface{}{"b": "y"}, map[string]interface{}{"b": "z"}}}, "", false}, // non-existent slice index
{"a.502.b", map[string]interface{}{"a": []interface{}{}}, "", false}, // non-existent slice index
{"a.b.503", map[string]interface{}{"a": map[string]interface{}{"b": []interface{}{"x", "y", "z"}}}, "", false}, // trailing, non-existent slice index
{"a.b", interface{}("a"), "", false}, // non-map, non-slice input
}
func TestExtractParameter(t *testing.T) {
for _, tt := range extractParameterTests {
value, err := ExtractParameterAsString(tt.s, tt.params)
if (err == nil) != tt.ok || value != tt.value {
t.Errorf("failed to extract parameter %q:\nexpected {value:%#v, ok:%#v},\ngot {value:%#v, err:%v}", tt.s, tt.value, tt.ok, value, err)
}
}
}
var argumentGetTests = []struct {
source, name string
headers, query, payload map[string]interface{}
request *http.Request
value string
ok bool
}{
{"header", "a", map[string]interface{}{"A": "z"}, nil, nil, nil, "z", true},
{"url", "a", nil, map[string]interface{}{"a": "z"}, nil, nil, "z", true},
{"payload", "a", nil, nil, map[string]interface{}{"a": "z"}, nil, "z", true},
{"request", "METHOD", nil, nil, map[string]interface{}{"a": "z"}, &http.Request{Method: "POST", RemoteAddr: "127.0.0.1:1234"}, "POST", true},
{"request", "remote-addr", nil, nil, map[string]interface{}{"a": "z"}, &http.Request{Method: "POST", RemoteAddr: "127.0.0.1:1234"}, "127.0.0.1:1234", true},
{"string", "a", nil, nil, map[string]interface{}{"a": "z"}, nil, "a", true},
// failures
{"header", "a", nil, map[string]interface{}{"a": "z"}, map[string]interface{}{"a": "z"}, nil, "", false}, // nil headers
{"url", "a", map[string]interface{}{"A": "z"}, nil, map[string]interface{}{"a": "z"}, nil, "", false}, // nil query
{"payload", "a", map[string]interface{}{"A": "z"}, map[string]interface{}{"a": "z"}, nil, nil, "", false}, // nil payload
{"foo", "a", map[string]interface{}{"A": "z"}, nil, nil, nil, "", false}, // invalid source
}
func TestArgumentGet(t *testing.T) {
for _, tt := range argumentGetTests {
a := Argument{tt.source, tt.name, "", false}
r := &Request{
Headers: tt.headers,
Query: tt.query,
Payload: tt.payload,
RawRequest: tt.request,
}
value, err := a.Get(r)
if (err == nil) != tt.ok || value != tt.value {
t.Errorf("failed to get {%q, %q}:\nexpected {value:%#v, ok:%#v},\ngot {value:%#v, err:%v}", tt.source, tt.name, tt.value, tt.ok, value, err)
}
}
}
var hookParseJSONParametersTests = []struct {
params []Argument
headers, query, payload map[string]interface{}
rheaders, rquery, rpayload map[string]interface{}
ok bool
}{
{[]Argument{Argument{"header", "a", "", false}}, map[string]interface{}{"A": `{"b": "y"}`}, nil, nil, map[string]interface{}{"A": map[string]interface{}{"b": "y"}}, nil, nil, true},
{[]Argument{Argument{"url", "a", "", false}}, nil, map[string]interface{}{"a": `{"b": "y"}`}, nil, nil, map[string]interface{}{"a": map[string]interface{}{"b": "y"}}, nil, true},
{[]Argument{Argument{"payload", "a", "", false}}, nil, nil, map[string]interface{}{"a": `{"b": "y"}`}, nil, nil, map[string]interface{}{"a": map[string]interface{}{"b": "y"}}, true},
{[]Argument{Argument{"header", "z", "", false}}, map[string]interface{}{"Z": `{}`}, nil, nil, map[string]interface{}{"Z": map[string]interface{}{}}, nil, nil, true},
// failures
{[]Argument{Argument{"header", "z", "", false}}, map[string]interface{}{"Z": ``}, nil, nil, map[string]interface{}{"Z": ``}, nil, nil, false}, // empty string
{[]Argument{Argument{"header", "y", "", false}}, map[string]interface{}{"X": `{}`}, nil, nil, map[string]interface{}{"X": `{}`}, nil, nil, false}, // missing parameter
{[]Argument{Argument{"string", "z", "", false}}, map[string]interface{}{"Z": ``}, nil, nil, map[string]interface{}{"Z": ``}, nil, nil, false}, // invalid argument source
}
func TestHookParseJSONParameters(t *testing.T) {
for _, tt := range hookParseJSONParametersTests {
h := &Hook{JSONStringParameters: tt.params}
r := &Request{
Headers: tt.headers,
Query: tt.query,
Payload: tt.payload,
}
err := h.ParseJSONParameters(r)
if (err == nil) != tt.ok || !reflect.DeepEqual(tt.headers, tt.rheaders) {
t.Errorf("failed to parse %v:\nexpected %#v, ok: %v\ngot %#v, ok: %v", tt.params, tt.rheaders, tt.ok, tt.headers, (err == nil))
}
}
}
var hookExtractCommandArgumentsTests = []struct {
exec string
args []Argument
headers, query, payload map[string]interface{}
value []string
ok bool
}{
{"test", []Argument{Argument{"header", "a", "", false}}, map[string]interface{}{"A": "z"}, nil, nil, []string{"test", "z"}, true},
// failures
{"fail", []Argument{Argument{"payload", "a", "", false}}, map[string]interface{}{"A": "z"}, nil, nil, []string{"fail", ""}, false},
}
func TestHookExtractCommandArguments(t *testing.T) {
for _, tt := range hookExtractCommandArgumentsTests {
h := &Hook{ExecuteCommand: tt.exec, PassArgumentsToCommand: tt.args}
r := &Request{
Headers: tt.headers,
Query: tt.query,
Payload: tt.payload,
}
value, err := h.ExtractCommandArguments(r)
if (err == nil) != tt.ok || !reflect.DeepEqual(value, tt.value) {
t.Errorf("failed to extract args {cmd=%q, args=%v}:\nexpected %#v, ok: %v\ngot %#v, ok: %v", tt.exec, tt.args, tt.value, tt.ok, value, (err == nil))
}
}
}
// Here we test the extraction of env variables when the user defined a hook
// with the "pass-environment-to-command" directive
// we test both cases where the name of the data is used as the name of the
// env key & the case where the hook definition sets the env var name to a
// fixed value using the envname construct like so::
// [
// {
// "id": "push",
// "execute-command": "bb2mm",
// "command-working-directory": "/tmp",
// "pass-environment-to-command":
// [
// {
// "source": "entire-payload",
// "envname": "PAYLOAD"
// },
// ]
// }
// ]
var hookExtractCommandArgumentsForEnvTests = []struct {
exec string
args []Argument
headers, query, payload map[string]interface{}
value []string
ok bool
}{
// successes
{
"test",
[]Argument{Argument{"header", "a", "", false}},
map[string]interface{}{"A": "z"}, nil, nil,
[]string{"HOOK_a=z"},
true,
},
{
"test",
[]Argument{Argument{"header", "a", "MYKEY", false}},
map[string]interface{}{"A": "z"}, nil, nil,
[]string{"MYKEY=z"},
true,
},
// failures
{
"fail",
[]Argument{Argument{"payload", "a", "", false}},
map[string]interface{}{"A": "z"}, nil, nil,
[]string{},
false,
},
}
func TestHookExtractCommandArgumentsForEnv(t *testing.T) {
for _, tt := range hookExtractCommandArgumentsForEnvTests {
h := &Hook{ExecuteCommand: tt.exec, PassEnvironmentToCommand: tt.args}
r := &Request{
Headers: tt.headers,
Query: tt.query,
Payload: tt.payload,
}
value, err := h.ExtractCommandArgumentsForEnv(r)
if (err == nil) != tt.ok || !reflect.DeepEqual(value, tt.value) {
t.Errorf("failed to extract args for env {cmd=%q, args=%v}:\nexpected %#v, ok: %v\ngot %#v, ok: %v", tt.exec, tt.args, tt.value, tt.ok, value, (err == nil))
}
}
}
var hooksLoadFromFileTests = []struct {
path string
asTemplate bool
ok bool
}{
{"../../hooks.json.example", false, true},
{"../../hooks.yaml.example", false, true},
{"../../hooks.json.tmpl.example", true, true},
{"../../hooks.yaml.tmpl.example", true, true},
{"", false, true},
// failures
{"missing.json", false, false},
}
func TestHooksLoadFromFile(t *testing.T) {
secret := `foo"123`
os.Setenv("XXXTEST_SECRET", secret)
for _, tt := range hooksLoadFromFileTests {
h := &Hooks{}
err := h.LoadFromFile(tt.path, tt.asTemplate)
if (err == nil) != tt.ok {
t.Errorf(err.Error())
}
}
}
func TestHooksTemplateLoadFromFile(t *testing.T) {
secret := `foo"123`
os.Setenv("XXXTEST_SECRET", secret)
for _, tt := range hooksLoadFromFileTests {
if !tt.asTemplate {
continue
}
h := &Hooks{}
err := h.LoadFromFile(tt.path, tt.asTemplate)
if (err == nil) != tt.ok {
t.Errorf(err.Error())
continue
}
s := (*h.Match("webhook").TriggerRule.And)[0].Match.Secret
if s != secret {
t.Errorf("Expected secret of %q, got %q", secret, s)
}
}
}
var hooksMatchTests = []struct {
id string
hooks Hooks
value *Hook
}{
{"a", Hooks{Hook{ID: "a"}}, &Hook{ID: "a"}},
{"X", Hooks{Hook{ID: "a"}}, new(Hook)},
}
func TestHooksMatch(t *testing.T) {
for _, tt := range hooksMatchTests {
value := tt.hooks.Match(tt.id)
if reflect.DeepEqual(reflect.ValueOf(value), reflect.ValueOf(tt.value)) {
t.Errorf("failed to match %q:\nexpected %#v,\ngot %#v", tt.id, tt.value, value)
}
}
}
var matchRuleTests = []struct {
typ, regex, secret, value, ipRange string
param Argument
headers, query, payload map[string]interface{}
body []byte
remoteAddr string
ok bool
err bool
}{
{"value", "", "", "z", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": "z"}, nil, nil, []byte{}, "", true, false},
{"regex", "^z", "", "z", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": "z"}, nil, nil, []byte{}, "", true, false},
{"payload-hmac-sha1", "", "secret", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": "b17e04cbb22afa8ffbff8796fc1894ed27badd9e"}, nil, nil, []byte(`{"a": "z"}`), "", true, false},
{"payload-hash-sha1", "", "secret", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": "b17e04cbb22afa8ffbff8796fc1894ed27badd9e"}, nil, nil, []byte(`{"a": "z"}`), "", true, false},
{"payload-hmac-sha256", "", "secret", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": "f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89"}, nil, nil, []byte(`{"a": "z"}`), "", true, false},
{"payload-hash-sha256", "", "secret", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": "f417af3a21bd70379b5796d5f013915e7029f62c580fb0f500f59a35a6f04c89"}, nil, nil, []byte(`{"a": "z"}`), "", true, false},
// failures
{"value", "", "", "X", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": "z"}, nil, nil, []byte{}, "", false, false},
{"regex", "^X", "", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": "z"}, nil, nil, []byte{}, "", false, false},
{"value", "", "2", "X", "", Argument{"header", "a", "", false}, map[string]interface{}{"Y": "z"}, nil, nil, []byte{}, "", false, true}, // reference invalid header
// errors
{"regex", "*", "", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": "z"}, nil, nil, []byte{}, "", false, true}, // invalid regex
{"payload-hmac-sha1", "", "secret", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": ""}, nil, nil, []byte{}, "", false, true}, // invalid hmac
{"payload-hash-sha1", "", "secret", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": ""}, nil, nil, []byte{}, "", false, true}, // invalid hmac
{"payload-hmac-sha256", "", "secret", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": ""}, nil, nil, []byte{}, "", false, true}, // invalid hmac
{"payload-hash-sha256", "", "secret", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": ""}, nil, nil, []byte{}, "", false, true}, // invalid hmac
{"payload-hmac-sha512", "", "secret", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": ""}, nil, nil, []byte{}, "", false, true}, // invalid hmac
{"payload-hash-sha512", "", "secret", "", "", Argument{"header", "a", "", false}, map[string]interface{}{"A": ""}, nil, nil, []byte{}, "", false, true}, // invalid hmac
// IP whitelisting, valid cases
{"ip-whitelist", "", "", "", "192.168.0.1/24", Argument{}, nil, nil, nil, []byte{}, "192.168.0.2:9000", true, false}, // valid IPv4, with range
{"ip-whitelist", "", "", "", "192.168.0.1/24", Argument{}, nil, nil, nil, []byte{}, "192.168.0.2:9000", true, false}, // valid IPv4, with range
{"ip-whitelist", "", "", "", "192.168.0.1", Argument{}, nil, nil, nil, []byte{}, "192.168.0.1:9000", true, false}, // valid IPv4, no range
{"ip-whitelist", "", "", "", "::1/24", Argument{}, nil, nil, nil, []byte{}, "[::1]:9000", true, false}, // valid IPv6, with range
{"ip-whitelist", "", "", "", "::1", Argument{}, nil, nil, nil, []byte{}, "[::1]:9000", true, false}, // valid IPv6, no range
// IP whitelisting, invalid cases
{"ip-whitelist", "", "", "", "192.168.0.1/a", Argument{}, nil, nil, nil, []byte{}, "192.168.0.2:9000", false, true}, // invalid IPv4, with range
{"ip-whitelist", "", "", "", "192.168.0.a", Argument{}, nil, nil, nil, []byte{}, "192.168.0.2:9000", false, true}, // invalid IPv4, no range
{"ip-whitelist", "", "", "", "192.168.0.1/24", Argument{}, nil, nil, nil, []byte{}, "192.168.0.a:9000", false, true}, // invalid IPv4 address
{"ip-whitelist", "", "", "", "::1/a", Argument{}, nil, nil, nil, []byte{}, "[::1]:9000", false, true}, // invalid IPv6, with range
{"ip-whitelist", "", "", "", "::z", Argument{}, nil, nil, nil, []byte{}, "[::1]:9000", false, true}, // invalid IPv6, no range
{"ip-whitelist", "", "", "", "::1/24", Argument{}, nil, nil, nil, []byte{}, "[::z]:9000", false, true}, // invalid IPv6 address
}
func TestMatchRule(t *testing.T) {
for i, tt := range matchRuleTests {
r := MatchRule{tt.typ, tt.regex, tt.secret, tt.value, tt.param, tt.ipRange}
req := &Request{
Headers: tt.headers,
Query: tt.query,
Payload: tt.payload,
Body: tt.body,
RawRequest: &http.Request{
RemoteAddr: tt.remoteAddr,
},
}
ok, err := r.Evaluate(req)
if ok != tt.ok || (err != nil) != tt.err {
t.Errorf("%d failed to match %#v:\nexpected ok: %#v, err: %v\ngot ok: %#v, err: %v", i, r, tt.ok, tt.err, ok, err)
}
}
}
var andRuleTests = []struct {
desc string // description of the test case
rule AndRule
headers, query, payload map[string]interface{}
body []byte
ok bool
err bool
}{
{
"(a=z, b=y): a=z && b=y",
AndRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", "", false}, ""}},
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", "", false}, ""}},
},
map[string]interface{}{"A": "z", "B": "y"}, nil, nil,
[]byte{},
true, false,
},
{
"(a=z, b=Y): a=z && b=y",
AndRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", "", false}, ""}},
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", "", false}, ""}},
},
map[string]interface{}{"A": "z", "B": "Y"}, nil, nil,
[]byte{},
false, false,
},
// Complex test to cover Rules.Evaluate
{
"(a=z, b=y, c=x, d=w=, e=X, f=X): a=z && (b=y && c=x) && (d=w || e=v) && !f=u",
AndRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", "", false}, ""}},
{
And: &AndRule{
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", "", false}, ""}},
{Match: &MatchRule{"value", "", "", "x", Argument{"header", "c", "", false}, ""}},
},
},
{
Or: &OrRule{
{Match: &MatchRule{"value", "", "", "w", Argument{"header", "d", "", false}, ""}},
{Match: &MatchRule{"value", "", "", "v", Argument{"header", "e", "", false}, ""}},
},
},
{
Not: &NotRule{
Match: &MatchRule{"value", "", "", "u", Argument{"header", "f", "", false}, ""},
},
},
},
map[string]interface{}{"A": "z", "B": "y", "C": "x", "D": "w", "E": "X", "F": "X"}, nil, nil,
[]byte{},
true, false,
},
{"empty rule", AndRule{{}}, nil, nil, nil, nil, false, false},
// failures
{
"invalid rule",
AndRule{{Match: &MatchRule{"value", "", "", "X", Argument{"header", "a", "", false}, ""}}},
map[string]interface{}{"Y": "z"}, nil, nil, nil,
false, true,
},
}
func TestAndRule(t *testing.T) {
for _, tt := range andRuleTests {
r := &Request{
Headers: tt.headers,
Query: tt.query,
Payload: tt.payload,
Body: tt.body,
}
ok, err := tt.rule.Evaluate(r)
if ok != tt.ok || (err != nil) != tt.err {
t.Errorf("failed to match %#v:\nexpected ok: %#v, err: %v\ngot ok: %#v, err: %v", tt.desc, tt.ok, tt.err, ok, err)
}
}
}
var orRuleTests = []struct {
desc string // description of the test case
rule OrRule
headers, query, payload map[string]interface{}
body []byte
ok bool
err bool
}{
{
"(a=z, b=X): a=z || b=y",
OrRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", "", false}, ""}},
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", "", false}, ""}},
},
map[string]interface{}{"A": "z", "B": "X"}, nil, nil,
[]byte{},
true, false,
},
{
"(a=X, b=y): a=z || b=y",
OrRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", "", false}, ""}},
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", "", false}, ""}},
},
map[string]interface{}{"A": "X", "B": "y"}, nil, nil,
[]byte{},
true, false,
},
{
"(a=Z, b=Y): a=z || b=y",
OrRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", "", false}, ""}},
{Match: &MatchRule{"value", "", "", "y", Argument{"header", "b", "", false}, ""}},
},
map[string]interface{}{"A": "Z", "B": "Y"}, nil, nil,
[]byte{},
false, false,
},
// failures
{
"missing parameter node",
OrRule{
{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", "", false}, ""}},
},
map[string]interface{}{"Y": "Z"}, nil, nil,
[]byte{},
false, false,
},
}
func TestOrRule(t *testing.T) {
for _, tt := range orRuleTests {
r := &Request{
Headers: tt.headers,
Query: tt.query,
Payload: tt.payload,
Body: tt.body,
}
ok, err := tt.rule.Evaluate(r)
if ok != tt.ok || (err != nil) != tt.err {
t.Errorf("%#v:\nexpected ok: %#v, err: %v\ngot ok: %#v err: %v", tt.desc, tt.ok, tt.err, ok, err)
}
}
}
var notRuleTests = []struct {
desc string // description of the test case
rule NotRule
headers, query, payload map[string]interface{}
body []byte
ok bool
err bool
}{
{"(a=z): !a=X", NotRule{Match: &MatchRule{"value", "", "", "X", Argument{"header", "a", "", false}, ""}}, map[string]interface{}{"A": "z"}, nil, nil, []byte{}, true, false},
{"(a=z): !a=z", NotRule{Match: &MatchRule{"value", "", "", "z", Argument{"header", "a", "", false}, ""}}, map[string]interface{}{"A": "z"}, nil, nil, []byte{}, false, false},
}
func TestNotRule(t *testing.T) {
for _, tt := range notRuleTests {
r := &Request{
Headers: tt.headers,
Query: tt.query,
Payload: tt.payload,
Body: tt.body,
}
ok, err := tt.rule.Evaluate(r)
if ok != tt.ok || (err != nil) != tt.err {
t.Errorf("failed to match %#v:\nexpected ok: %#v, err: %v\ngot ok: %#v, err: %v", tt.rule, tt.ok, tt.err, ok, err)
}
}
}
func TestCompare(t *testing.T) {
for _, tt := range []struct {
a, b string
ok bool
}{
{"abcd", "abcd", true},
{"zyxw", "abcd", false},
} {
if ok := compare(tt.a, tt.b); ok != tt.ok {
t.Errorf("compare failed for %q and %q: got %v\n", tt.a, tt.b, ok)
}
}
}
+119
View File
@@ -0,0 +1,119 @@
package hook
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"net/url"
"unicode"
"github.com/clbanning/mxj"
)
// Request represents a webhook request.
type Request struct {
// The request ID set by the RequestID middleware.
ID string
// The Content-Type of the request.
ContentType string
// The raw request body.
Body []byte
// Headers is a map of the parsed headers.
Headers map[string]interface{}
// Query is a map of the parsed URL query values.
Query map[string]interface{}
// Payload is a map of the parsed payload.
Payload map[string]interface{}
// The underlying HTTP request.
RawRequest *http.Request
// Treat signature errors as simple validate failures.
AllowSignatureErrors bool
}
func (r *Request) ParseJSONPayload() error {
decoder := json.NewDecoder(bytes.NewReader(r.Body))
decoder.UseNumber()
var firstChar byte
for i := 0; i < len(r.Body); i++ {
if unicode.IsSpace(rune(r.Body[i])) {
continue
}
firstChar = r.Body[i]
break
}
if firstChar == byte('[') {
var arrayPayload interface{}
err := decoder.Decode(&arrayPayload)
if err != nil {
return fmt.Errorf("error parsing JSON array payload %+v", err)
}
r.Payload = make(map[string]interface{}, 1)
r.Payload["root"] = arrayPayload
} else {
err := decoder.Decode(&r.Payload)
if err != nil {
return fmt.Errorf("error parsing JSON payload %+v", err)
}
}
return nil
}
func (r *Request) ParseHeaders(headers map[string][]string) {
r.Headers = make(map[string]interface{}, len(headers))
for k, v := range headers {
if len(v) > 0 {
r.Headers[k] = v[0]
}
}
}
func (r *Request) ParseQuery(query map[string][]string) {
r.Query = make(map[string]interface{}, len(query))
for k, v := range query {
if len(v) > 0 {
r.Query[k] = v[0]
}
}
}
func (r *Request) ParseFormPayload() error {
fd, err := url.ParseQuery(string(r.Body))
if err != nil {
return fmt.Errorf("error parsing form payload %+v", err)
}
r.Payload = make(map[string]interface{}, len(fd))
for k, v := range fd {
if len(v) > 0 {
r.Payload[k] = v[0]
}
}
return nil
}
func (r *Request) ParseXMLPayload() error {
var err error
r.Payload, err = mxj.NewMapXmlReader(bytes.NewReader(r.Body))
if err != nil {
return fmt.Errorf("error parsing XML payload: %+v", err)
}
return nil
}
+105
View File
@@ -0,0 +1,105 @@
package middleware
// Derived from from the Goa project, MIT Licensed
// https://github.com/goadesign/goa/blob/v3/http/middleware/debug.go
import (
"bufio"
"bytes"
"fmt"
"io"
"net"
"net/http"
"net/http/httputil"
"sort"
"strings"
)
// responseDupper tees the response to a buffer and a response writer.
type responseDupper struct {
http.ResponseWriter
Buffer *bytes.Buffer
Status int
}
// Dumper returns a debug middleware which prints detailed information about
// incoming requests and outgoing responses including all headers, parameters
// and bodies.
func Dumper(w io.Writer) func(http.Handler) http.Handler {
return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
buf := &bytes.Buffer{}
// Request ID
rid := r.Context().Value(RequestIDKey)
// Dump request
bd, err := httputil.DumpRequest(r, true)
if err != nil {
buf.WriteString(fmt.Sprintf("[%s] Error dumping request for debugging: %s\n", rid, err))
}
sc := bufio.NewScanner(bytes.NewBuffer(bd))
sc.Split(bufio.ScanLines)
for sc.Scan() {
buf.WriteString(fmt.Sprintf("> [%s] ", rid))
buf.WriteString(sc.Text() + "\n")
}
w.Write(buf.Bytes())
buf.Reset()
// Dump Response
dupper := &responseDupper{ResponseWriter: rw, Buffer: &bytes.Buffer{}}
h.ServeHTTP(dupper, r)
// Response Status
buf.WriteString(fmt.Sprintf("< [%s] %d %s\n", rid, dupper.Status, http.StatusText(dupper.Status)))
// Response Headers
keys := make([]string, len(dupper.Header()))
i := 0
for k := range dupper.Header() {
keys[i] = k
i++
}
sort.Strings(keys)
for _, k := range keys {
buf.WriteString(fmt.Sprintf("< [%s] %s: %s\n", rid, k, strings.Join(dupper.Header()[k], ", ")))
}
// Response Body
if dupper.Buffer.Len() > 0 {
buf.WriteString(fmt.Sprintf("< [%s]\n", rid))
sc = bufio.NewScanner(dupper.Buffer)
sc.Split(bufio.ScanLines)
for sc.Scan() {
buf.WriteString(fmt.Sprintf("< [%s] ", rid))
buf.WriteString(sc.Text() + "\n")
}
}
w.Write(buf.Bytes())
})
}
}
// Write writes the data to the buffer and connection as part of an HTTP reply.
func (r *responseDupper) Write(b []byte) (int, error) {
r.Buffer.Write(b)
return r.ResponseWriter.Write(b)
}
// WriteHeader records the status and sends an HTTP response header with status code.
func (r *responseDupper) WriteHeader(s int) {
r.Status = s
r.ResponseWriter.WriteHeader(s)
}
// Hijack supports the http.Hijacker interface.
func (r *responseDupper) Hijack() (net.Conn, *bufio.ReadWriter, error) {
if hijacker, ok := r.ResponseWriter.(http.Hijacker); ok {
return hijacker.Hijack()
}
return nil, nil, fmt.Errorf("dumper middleware: inner ResponseWriter cannot be hijacked: %T", r.ResponseWriter)
}
+59
View File
@@ -0,0 +1,59 @@
package middleware
import (
"bytes"
"fmt"
"log"
"net/http"
"time"
"github.com/dustin/go-humanize"
"github.com/go-chi/chi/middleware"
)
// Logger is a middleware that logs useful data about each HTTP request.
type Logger struct {
Logger middleware.LoggerInterface
}
// NewLogger creates a new RequestLogger Handler.
func NewLogger() func(next http.Handler) http.Handler {
return middleware.RequestLogger(&Logger{})
}
// NewLogEntry creates a new LogEntry for the request.
func (l *Logger) NewLogEntry(r *http.Request) middleware.LogEntry {
e := &LogEntry{
req: r,
buf: &bytes.Buffer{},
}
return e
}
// LogEntry represents an individual log entry.
type LogEntry struct {
*Logger
req *http.Request
buf *bytes.Buffer
}
// Write constructs and writes the final log entry.
func (l *LogEntry) Write(status, totalBytes int, elapsed time.Duration) {
rid := GetReqID(l.req.Context())
if rid != "" {
fmt.Fprintf(l.buf, "[%s] ", rid)
}
fmt.Fprintf(l.buf, "%03d | %s | %s | ", status, humanize.IBytes(uint64(totalBytes)), elapsed)
l.buf.WriteString(l.req.Host + " | " + l.req.Method + " " + l.req.RequestURI)
log.Print(l.buf.String())
}
// Panic prints the call stack for a panic.
func (l *LogEntry) Panic(v interface{}, stack []byte) {
e := l.NewLogEntry(l.req).(*LogEntry)
fmt.Fprintf(e.buf, "panic: %#v", v)
log.Print(e.buf.String())
log.Print(string(stack))
}
+98
View File
@@ -0,0 +1,98 @@
package middleware
// Derived from Goa project, MIT Licensed
// https://github.com/goadesign/goa/blob/v3/http/middleware/requestid.go
import (
"context"
"net/http"
"github.com/gofrs/uuid"
)
// Key to use when setting the request ID.
type ctxKeyRequestID int
// RequestIDKey is the key that holds the unique request ID in a request context.
const RequestIDKey ctxKeyRequestID = 0
// RequestID is a middleware that injects a request ID into the context of each
// request.
func RequestID(options ...RequestIDOption) func(http.Handler) http.Handler {
o := newRequestIDOptions(options...)
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var id string
if o.UseRequestID() {
id = r.Header.Get("X-Request-Id")
if o.requestIDLimit > 0 && len(id) > o.requestIDLimit {
id = id[:o.requestIDLimit]
}
}
if id == "" {
id = uuid.Must(uuid.NewV4()).String()[:6]
}
ctx = context.WithValue(ctx, RequestIDKey, id)
next.ServeHTTP(w, r.WithContext(ctx))
})
}
}
// GetReqID returns a request ID from the given context if one is present.
// Returns the empty string if a request ID cannot be found.
func GetReqID(ctx context.Context) string {
if ctx == nil {
return ""
}
if reqID, ok := ctx.Value(RequestIDKey).(string); ok {
return reqID
}
return ""
}
func UseXRequestIDHeaderOption(f bool) RequestIDOption {
return func(o *RequestIDOptions) *RequestIDOptions {
o.useXRequestID = f
return o
}
}
func XRequestIDLimitOption(limit int) RequestIDOption {
return func(o *RequestIDOptions) *RequestIDOptions {
o.requestIDLimit = limit
return o
}
}
type (
RequestIDOption func(*RequestIDOptions) *RequestIDOptions
RequestIDOptions struct {
// useXRequestID enabled the use of the X-Request-Id request header as
// the request ID.
useXRequestID bool
// requestIDLimit is the maximum length of the X-Request-Id header
// allowed. Values longer than this value are truncated. Zero value
// means no limit.
requestIDLimit int
}
)
func newRequestIDOptions(options ...RequestIDOption) *RequestIDOptions {
o := new(RequestIDOptions)
for _, opt := range options {
o = opt(o)
}
return o
}
func (o *RequestIDOptions) UseRequestID() bool {
return o.useXRequestID
}
+4
View File
@@ -0,0 +1,4 @@
Package pidfile is derived from github.com/moby/moby/pkg/pidfile.
Moby is licensed under the Apache License, Version 2.0.
Copyright 2012-2017 Docker, Inc.
+11
View File
@@ -0,0 +1,11 @@
// +build !windows
package pidfile
import "os"
// MkdirAll creates a directory named path along with any necessary parents,
// with permission specified by attribute perm for all dir created.
func MkdirAll(path string, perm os.FileMode) error {
return os.MkdirAll(path, perm)
}
+109
View File
@@ -0,0 +1,109 @@
// +build windows
package pidfile
import (
"os"
"regexp"
"syscall"
"unsafe"
"golang.org/x/sys/windows"
)
// MkdirAll implementation that is volume path aware for Windows. It can be used
// as a drop-in replacement for os.MkdirAll()
func MkdirAll(path string, _ os.FileMode) error {
return mkdirall(path, false, "")
}
// mkdirall is a custom version of os.MkdirAll modified for use on Windows
// so that it is both volume path aware, and can create a directory with
// a DACL.
func mkdirall(path string, applyACL bool, sddl string) error {
if re := regexp.MustCompile(`^\\\\\?\\Volume{[a-z0-9-]+}$`); re.MatchString(path) {
return nil
}
// The rest of this method is largely copied from os.MkdirAll and should be kept
// as-is to ensure compatibility.
// Fast path: if we can tell whether path is a directory or file, stop with success or error.
dir, err := os.Stat(path)
if err == nil {
if dir.IsDir() {
return nil
}
return &os.PathError{
Op: "mkdir",
Path: path,
Err: syscall.ENOTDIR,
}
}
// Slow path: make sure parent exists and then call Mkdir for path.
i := len(path)
for i > 0 && os.IsPathSeparator(path[i-1]) { // Skip trailing path separator.
i--
}
j := i
for j > 0 && !os.IsPathSeparator(path[j-1]) { // Scan backward over element.
j--
}
if j > 1 {
// Create parent
err = mkdirall(path[0:j-1], false, sddl)
if err != nil {
return err
}
}
// Parent now exists; invoke os.Mkdir or mkdirWithACL and use its result.
if applyACL {
err = mkdirWithACL(path, sddl)
} else {
err = os.Mkdir(path, 0)
}
if err != nil {
// Handle arguments like "foo/." by
// double-checking that directory doesn't exist.
dir, err1 := os.Lstat(path)
if err1 == nil && dir.IsDir() {
return nil
}
return err
}
return nil
}
// mkdirWithACL creates a new directory. If there is an error, it will be of
// type *PathError. .
//
// This is a modified and combined version of os.Mkdir and windows.Mkdir
// in golang to cater for creating a directory am ACL permitting full
// access, with inheritance, to any subfolder/file for Built-in Administrators
// and Local System.
func mkdirWithACL(name string, sddl string) error {
sa := windows.SecurityAttributes{Length: 0}
sd, err := windows.SecurityDescriptorFromString(sddl)
if err != nil {
return &os.PathError{Op: "mkdir", Path: name, Err: err}
}
sa.Length = uint32(unsafe.Sizeof(sa))
sa.InheritHandle = 1
sa.SecurityDescriptor = sd
namep, err := windows.UTF16PtrFromString(name)
if err != nil {
return &os.PathError{Op: "mkdir", Path: name, Err: err}
}
e := windows.CreateDirectory(namep, &sa)
if e != nil {
return &os.PathError{Op: "mkdir", Path: name, Err: e}
}
return nil
}
+51
View File
@@ -0,0 +1,51 @@
// Package pidfile provides structure and helper functions to create and remove
// PID file. A PID file is usually a file used to store the process ID of a
// running process.
package pidfile
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"strconv"
"strings"
)
// PIDFile is a file used to store the process ID of a running process.
type PIDFile struct {
path string
}
func checkPIDFileAlreadyExists(path string) error {
if pidByte, err := ioutil.ReadFile(path); err == nil {
pidString := strings.TrimSpace(string(pidByte))
if pid, err := strconv.Atoi(pidString); err == nil {
if processExists(pid) {
return fmt.Errorf("pid file found, ensure webhook is not running or delete %s", path)
}
}
}
return nil
}
// New creates a PIDfile using the specified path.
func New(path string) (*PIDFile, error) {
if err := checkPIDFileAlreadyExists(path); err != nil {
return nil, err
}
// Note MkdirAll returns nil if a directory already exists
if err := MkdirAll(filepath.Dir(path), os.FileMode(0o755)); err != nil {
return nil, err
}
if err := ioutil.WriteFile(path, []byte(fmt.Sprintf("%d", os.Getpid())), 0o600); err != nil {
return nil, err
}
return &PIDFile{path: path}, nil
}
// Remove removes the PIDFile.
func (file PIDFile) Remove() error {
return os.Remove(file.path)
}
+14
View File
@@ -0,0 +1,14 @@
// +build darwin
package pidfile
import (
"golang.org/x/sys/unix"
)
func processExists(pid int) bool {
// OS X does not have a proc filesystem.
// Use kill -0 pid to judge if the process exists.
err := unix.Kill(pid, 0)
return err == nil
}
+38
View File
@@ -0,0 +1,38 @@
package pidfile
import (
"io/ioutil"
"os"
"path/filepath"
"testing"
)
func TestNewAndRemove(t *testing.T) {
dir, err := ioutil.TempDir(os.TempDir(), "test-pidfile")
if err != nil {
t.Fatal("Could not create test directory")
}
path := filepath.Join(dir, "testfile")
file, err := New(path)
if err != nil {
t.Fatal("Could not create test file", err)
}
_, err = New(path)
if err == nil {
t.Fatal("Test file creation not blocked")
}
if err := file.Remove(); err != nil {
t.Fatal("Could not delete created test file")
}
}
func TestRemoveInvalidPath(t *testing.T) {
file := PIDFile{path: filepath.Join("foo", "bar")}
if err := file.Remove(); err == nil {
t.Fatal("Non-existing file doesn't give an error on delete")
}
}
+16
View File
@@ -0,0 +1,16 @@
// +build !windows,!darwin
package pidfile
import (
"os"
"path/filepath"
"strconv"
)
func processExists(pid int) bool {
if _, err := os.Stat(filepath.Join("/proc", strconv.Itoa(pid))); err == nil {
return true
}
return false
}
+25
View File
@@ -0,0 +1,25 @@
package pidfile
import (
"golang.org/x/sys/windows"
)
const (
processQueryLimitedInformation = 0x1000
stillActive = 259
)
func processExists(pid int) bool {
h, err := windows.OpenProcess(processQueryLimitedInformation, false, uint32(pid))
if err != nil {
return false
}
var c uint32
err = windows.GetExitCodeProcess(h, &c)
windows.Close(h)
if err != nil {
return c == stillActive
}
return true
}
+21 -3
View File
@@ -14,6 +14,9 @@ func setupSignals() {
signals = make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGUSR1)
signal.Notify(signals, syscall.SIGHUP)
signal.Notify(signals, syscall.SIGTERM)
signal.Notify(signals, os.Interrupt)
go watchForSignals()
}
@@ -23,11 +26,26 @@ func watchForSignals() {
for {
sig := <-signals
if sig == syscall.SIGUSR1 {
switch sig {
case syscall.SIGUSR1:
log.Println("caught USR1 signal")
reloadAllHooks()
reloadHooks()
} else {
case syscall.SIGHUP:
log.Println("caught HUP signal")
reloadAllHooks()
case os.Interrupt, syscall.SIGTERM:
log.Printf("caught %s signal; exiting\n", sig)
if pidFile != nil {
err := pidFile.Remove()
if err != nil {
log.Print(err)
}
}
os.Exit(0)
default:
log.Printf("caught unhandled signal %+v\n", sig)
}
}
+11
View File
@@ -5,6 +5,7 @@ package main
import (
"fmt"
"os"
"strconv"
"strings"
)
@@ -23,4 +24,14 @@ func main() {
if len(env) > 0 {
fmt.Printf("env: %s\n", strings.Join(env, " "))
}
if (len(os.Args) > 1) && (strings.HasPrefix(os.Args[1], "exit=")) {
exit_code_str := os.Args[1][5:]
exit_code, err := strconv.Atoi(exit_code_str)
if err != nil {
fmt.Printf("Exit code %s not an int!", exit_code_str)
os.Exit(-1)
}
os.Exit(exit_code)
}
}
+419 -1
View File
@@ -3,6 +3,7 @@
"id": "github",
"execute-command": "{{ .Hookecho }}",
"command-working-directory": "/",
"http-methods": ["Post "],
"include-command-output-in-response": true,
"trigger-rule-mismatch-http-response-code": 400,
"pass-environment-to-command":
@@ -30,7 +31,7 @@
{
"match":
{
"type": "payload-hash-sha1",
"type": "payload-hmac-sha1",
"secret": "mysecret",
"parameter":
{
@@ -54,6 +55,149 @@
]
}
},
{
"id": "github-multi-sig",
"execute-command": "{{ .Hookecho }}",
"command-working-directory": "/",
"http-methods": ["Post "],
"include-command-output-in-response": true,
"trigger-rule-mismatch-http-response-code": 400,
"trigger-signature-soft-failures": true,
"pass-environment-to-command":
[
{
"source": "payload",
"name": "head_commit.timestamp"
}
],
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "head_commit.id"
},
{
"source": "payload",
"name": "head_commit.author.email"
}
],
"trigger-rule":
{
"and":
[
"or":
[
{
"match":
{
"type": "payload-hmac-sha1",
"secret": "mysecretFAIL",
"parameter":
{
"source": "header",
"name": "X-Hub-Signature"
}
}
},
{
"match":
{
"type": "payload-hmac-sha1",
"secret": "mysecret",
"parameter":
{
"source": "header",
"name": "X-Hub-Signature"
}
}
}
],
{
"match":
{
"type": "value",
"value": "refs/heads/master",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
]
}
},
{
"id": "github-multi-sig-fail",
"execute-command": "{{ .Hookecho }}",
"command-working-directory": "/",
"http-methods": ["Post "],
"include-command-output-in-response": true,
"trigger-rule-mismatch-http-response-code": 400,
"pass-environment-to-command":
[
{
"source": "payload",
"name": "head_commit.timestamp"
}
],
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "head_commit.id"
},
{
"source": "payload",
"name": "head_commit.author.email"
}
],
"trigger-rule":
{
"and":
[
"or":
[
{
"match":
{
"type": "payload-hmac-sha1",
"secret": "mysecretFAIL",
"parameter":
{
"source": "header",
"name": "X-Hub-Signature"
}
}
},
{
"match":
{
"type": "payload-hmac-sha1",
"secret": "mysecret",
"parameter":
{
"source": "header",
"name": "X-Hub-Signature"
}
}
}
],
{
"match":
{
"type": "value",
"value": "refs/heads/master",
"parameter":
{
"source": "payload",
"name": "ref"
}
}
}
]
}
},
{
"id": "bitbucket",
"execute-command": "{{ .Hookecho }}",
@@ -136,5 +280,279 @@
}
}
}
},
{
"id": "xml",
"execute-command": "{{ .Hookecho }}",
"command-working-directory": "/",
"response-message": "success",
"trigger-rule": {
"and": [
{
"match": {
"type": "value",
"parameter": {
"source": "payload",
"name": "app.users.user.0.-name"
},
"value": "Jeff"
}
},
{
"match": {
"type": "value",
"parameter": {
"source": "payload",
"name": "app.messages.message.#text"
},
"value": "Hello!!"
}
},
],
}
},
{
"id": "txt-raw",
"execute-command": "{{ .Hookecho }}",
"command-working-directory": "/",
"include-command-output-in-response": true,
"pass-arguments-to-command": [
{
"source": "raw-request-body"
}
]
},
{
"id": "sendgrid",
"execute-command": "{{ .Hookecho }}",
"command-working-directory": "/",
"response-message": "success",
"trigger-rule": {
"match": {
"type": "value",
"parameter": {
"source": "payload",
"name": "root.0.event"
},
"value": "processed"
}
}
},
{
"id": "sendgrid/dir",
"execute-command": "{{ .Hookecho }}",
"command-working-directory": "/",
"response-message": "success",
"trigger-rule": {
"match": {
"type": "value",
"parameter": {
"source": "payload",
"name": "root.0.event"
},
"value": "it worked!"
}
}
},
{
"id": "plex",
"execute-command": "{{ .Hookecho }}",
"command-working-directory": "/",
"response-message": "success",
"parse-parameters-as-json": [
{
"source": "payload",
"name": "payload"
}
],
"trigger-rule":
{
"match":
{
"type": "value",
"parameter": {
"source": "payload",
"name": "payload.event"
},
"value": "media.play"
}
}
},
{
"id": "capture-command-output-on-success-not-by-default",
"pass-arguments-to-command": [
{
"source": "string",
"name": "exit=0"
}
],
"execute-command": "{{ .Hookecho }}"
},
{
"id": "capture-command-output-on-success-yes-with-flag",
"pass-arguments-to-command": [
{
"source": "string",
"name": "exit=0"
}
],
"execute-command": "{{ .Hookecho }}",
"include-command-output-in-response": true
},
{
"id": "capture-command-output-on-error-not-by-default",
"pass-arguments-to-command": [
{
"source": "string",
"name": "exit=1"
}
],
"execute-command": "{{ .Hookecho }}",
"include-command-output-in-response": true
},
{
"id": "capture-command-output-on-error-yes-with-extra-flag",
"pass-arguments-to-command": [
{
"source": "string",
"name": "exit=1"
}
],
"execute-command": "{{ .Hookecho }}",
"include-command-output-in-response": true,
"include-command-output-in-response-on-error": true
},
{
"id": "request-source",
"pass-arguments-to-command": [
{
"source": "request",
"name": "method"
},
{
"source": "request",
"name": "remote-addr"
}
],
"execute-command": "{{ .Hookecho }}",
"include-command-output-in-response": true
},
{
"id": "static-params-ok",
"execute-command": "{{ .Hookecho }}",
"response-message": "success",
"include-command-output-in-response": true,
"pass-arguments-to-command": [
{
"source": "string",
"name": "passed"
}
],
},
{
"id": "warn-on-space",
"execute-command": "{{ .Hookecho }} foo",
"response-message": "success",
"include-command-output-in-response": true,
"pass-arguments-to-command": [
{
"source": "string",
"name": "passed"
}
],
},
{
"id": "issue-471",
"execute-command": "{{ .Hookecho }}",
"response-message": "success",
"trigger-rule":
{
"or":
[
{
"match":
{
"parameter":
{
"source": "payload",
"name": "foo"
},
"type": "value",
"value": "bar"
}
},
{
"match":
{
"parameter":
{
"source": "payload",
"name": "exists"
},
"type": "value",
"value": 1
}
}
]
}
},
{
"id": "issue-471-and",
"execute-command": "{{ .Hookecho }}",
"response-message": "success",
"trigger-rule":
{
"and":
[
{
"match":
{
"parameter":
{
"source": "payload",
"name": "foo"
},
"type": "value",
"value": "bar"
}
},
{
"match":
{
"parameter":
{
"source": "payload",
"name": "exists"
},
"type": "value",
"value": 1
}
}
]
}
},
{
"id": "empty-payload-signature",
"execute-command": "{{ .Hookecho }}",
"command-working-directory": "/",
"include-command-output-in-response": true,
"trigger-rule":
{
"and":
[
{
"match":
{
"type": "payload-hmac-sha1",
"secret": "mysecret",
"parameter":
{
"source": "header",
"name": "X-Hub-Signature"
}
}
}
]
}
}
]
+316
View File
@@ -0,0 +1,316 @@
- id: github
http-methods:
- "Post "
trigger-rule:
and:
- match:
parameter:
source: header
name: X-Hub-Signature
secret: mysecret
type: payload-hmac-sha1
- match:
parameter:
source: payload
name: ref
type: value
value: refs/heads/master
include-command-output-in-response: true
trigger-rule-mismatch-http-response-code: 400
execute-command: '{{ .Hookecho }}'
pass-arguments-to-command:
- source: payload
name: head_commit.id
- source: payload
name: head_commit.author.email
pass-environment-to-command:
- source: payload
name: head_commit.timestamp
command-working-directory: /
- id: github-multi-sig
http-methods:
- "Post "
trigger-rule:
and:
- or:
- match:
parameter:
source: header
name: X-Hub-Signature
secret: mysecretFAIL
type: payload-hmac-sha1
- match:
parameter:
source: header
name: X-Hub-Signature
secret: mysecret
type: payload-hmac-sha1
- match:
parameter:
source: payload
name: ref
type: value
value: refs/heads/master
include-command-output-in-response: true
trigger-rule-mismatch-http-response-code: 400
trigger-signature-soft-failures: true
execute-command: '{{ .Hookecho }}'
pass-arguments-to-command:
- source: payload
name: head_commit.id
- source: payload
name: head_commit.author.email
pass-environment-to-command:
- source: payload
name: head_commit.timestamp
command-working-directory: /
- id: github-multi-sig-fail
http-methods:
- "Post "
trigger-rule:
and:
- or:
- match:
parameter:
source: header
name: X-Hub-Signature
secret: mysecretFAIL
type: payload-hmac-sha1
- match:
parameter:
source: header
name: X-Hub-Signature
secret: mysecret
type: payload-hmac-sha1
- match:
parameter:
source: payload
name: ref
type: value
value: refs/heads/master
include-command-output-in-response: true
trigger-rule-mismatch-http-response-code: 400
execute-command: '{{ .Hookecho }}'
pass-arguments-to-command:
- source: payload
name: head_commit.id
- source: payload
name: head_commit.author.email
pass-environment-to-command:
- source: payload
name: head_commit.timestamp
command-working-directory: /
- id: bitbucket
trigger-rule:
and:
- match:
parameter:
source: payload
name: payload.canon_url
type: value
value: https://bitbucket.org
- match:
parameter:
source: payload
name: payload.repository.absolute_url
type: value
value: /webhook/testing/
- match:
parameter:
source: payload
name: payload.commits.0.branch
type: value
value: master
parse-parameters-as-json:
- source: payload
name: payload
trigger-rule-mismatch-http-response-code: 999
execute-command: '{{ .Hookecho }}'
response-message: success
include-command-output-in-response: false
command-working-directory: /
- id: gitlab
trigger-rule:
match:
parameter:
source: payload
name: ref
type: value
value: refs/heads/master
pass-arguments-to-command:
- source: payload
name: commits.0.id
- source: payload
name: user_name
- source: payload
name: user_email
execute-command: '{{ .Hookecho }}'
response-message: success
include-command-output-in-response: true
command-working-directory: /
- id: xml
execute-command: '{{ .Hookecho }}'
command-working-directory: /
response-message: success
trigger-rule:
and:
- match:
type: value
parameter:
source: payload
name: app.users.user.0.-name
value: Jeff
- match:
type: value
parameter:
source: payload
name: "app.messages.message.#text"
value: "Hello!!"
- id: txt-raw
execute-command: '{{ .Hookecho }}'
command-working-directory: /
include-command-output-in-response: true
pass-arguments-to-command:
- source: raw-request-body
- id: sendgrid
execute-command: '{{ .Hookecho }}'
command-working-directory: /
response-message: success
trigger-rule:
match:
type: value
parameter:
source: payload
name: root.0.event
value: processed
- id: sendgrid/dir
execute-command: '{{ .Hookecho }}'
command-working-directory: /
response-message: success
trigger-rule:
match:
type: value
parameter:
source: payload
name: root.0.event
value: it worked!
- id: plex
trigger-rule:
match:
type: value
parameter:
source: payload
name: payload.event
value: media.play
parse-parameters-as-json:
- source: payload
name: payload
execute-command: '{{ .Hookecho }}'
response-message: success
command-working-directory: /
- id: capture-command-output-on-success-not-by-default
pass-arguments-to-command:
- source: string
name: exit=0
execute-command: '{{ .Hookecho }}'
- id: capture-command-output-on-success-yes-with-flag
pass-arguments-to-command:
- source: string
name: exit=0
execute-command: '{{ .Hookecho }}'
include-command-output-in-response: true
- id: capture-command-output-on-error-not-by-default
pass-arguments-to-command:
- source: string
name: exit=1
execute-command: '{{ .Hookecho }}'
include-command-output-in-response: true
- id: capture-command-output-on-error-yes-with-extra-flag
pass-arguments-to-command:
- source: string
name: exit=1
execute-command: '{{ .Hookecho }}'
include-command-output-in-response: true
include-command-output-in-response-on-error: true
- id: request-source
pass-arguments-to-command:
- source: request
name: method
- source: request
name: remote-addr
execute-command: '{{ .Hookecho }}'
include-command-output-in-response: true
- id: static-params-ok
execute-command: '{{ .Hookecho }}'
include-command-output-in-response: true
pass-arguments-to-command:
- source: string
name: passed
- id: warn-on-space
execute-command: '{{ .Hookecho }} foo'
include-command-output-in-response: true
- id: issue-471
execute-command: '{{ .Hookecho }}'
response-message: success
trigger-rule:
or:
- match:
parameter:
source: payload
name: foo
type: value
value: bar
- match:
parameter:
source: payload
name: exists
type: value
value: 1
- id: issue-471-and
execute-command: '{{ .Hookecho }}'
response-message: success
trigger-rule:
and:
- match:
parameter:
source: payload
name: foo
type: value
value: bar
- match:
parameter:
source: payload
name: exists
type: value
value: 1
- id: empty-payload-signature
include-command-output-in-response: true
execute-command: '{{ .Hookecho }}'
command-working-directory: /
trigger-rule:
and:
- match:
parameter:
source: header
name: X-Hub-Signature
secret: mysecret
type: payload-hmac-sha1
+85
View File
@@ -0,0 +1,85 @@
package main
import (
"crypto/tls"
"io"
"log"
"strings"
)
func writeTLSSupportedCipherStrings(w io.Writer, min uint16) error {
for _, c := range tls.CipherSuites() {
var found bool
for _, v := range c.SupportedVersions {
if v >= min {
found = true
}
}
if !found {
continue
}
_, err := w.Write([]byte(c.Name + "\n"))
if err != nil {
return err
}
}
return nil
}
// getTLSMinVersion converts a version string into a TLS version ID.
func getTLSMinVersion(v string) uint16 {
switch v {
case "1.0":
return tls.VersionTLS10
case "1.1":
return tls.VersionTLS11
case "1.2", "":
return tls.VersionTLS12
case "1.3":
return tls.VersionTLS13
default:
log.Fatalln("error: unknown minimum TLS version:", v)
return 0
}
}
// getTLSCipherSuites converts a comma separated list of cipher suites into a
// slice of TLS cipher suite IDs.
func getTLSCipherSuites(v string) []uint16 {
supported := tls.CipherSuites()
if v == "" {
suites := make([]uint16, len(supported))
for _, cs := range supported {
suites = append(suites, cs.ID)
}
return suites
}
var found bool
txts := strings.Split(v, ",")
suites := make([]uint16, len(txts))
for _, want := range txts {
found = false
for _, cs := range supported {
if want == cs.Name {
suites = append(suites, cs.ID)
found = true
}
}
if !found {
log.Fatalln("error: unknown TLS cipher suite:", want)
}
}
return suites
}
+7
View File
@@ -0,0 +1,7 @@
*.exe
*.exe~
*.dll
*.so
*.dylib
*.test
*.out
+3
View File
@@ -0,0 +1,3 @@
language: go
go:
- 1.13.x
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2019 Anton Medvedev
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+169
View File
@@ -0,0 +1,169 @@
# Expr
[![Build Status](https://travis-ci.org/antonmedv/expr.svg?branch=master)](https://travis-ci.org/antonmedv/expr)
[![Go Report Card](https://goreportcard.com/badge/github.com/antonmedv/expr)](https://goreportcard.com/report/github.com/antonmedv/expr)
[![GoDoc](https://godoc.org/github.com/antonmedv/expr?status.svg)](https://godoc.org/github.com/antonmedv/expr)
<img src="docs/images/logo-small.png" width="150" alt="expr logo" align="right">
**Expr** package provides an engine that can compile and evaluate expressions.
An expression is a one-liner that returns a value (mostly, but not limited to, booleans).
It is designed for simplicity, speed and safety.
The purpose of the package is to allow users to use expressions inside configuration for more complex logic.
It is a perfect candidate for the foundation of a _business rule engine_.
The idea is to let configure things in a dynamic way without recompile of a program:
```coffeescript
# Get the special price if
user.Group in ["good_customers", "collaborator"]
# Promote article to the homepage when
len(article.Comments) > 100 and article.Category not in ["misc"]
# Send an alert when
product.Stock < 15
```
## Features
* Seamless integration with Go (no need to redefine types)
* Static typing ([example](https://godoc.org/github.com/antonmedv/expr#example-Env)).
```go
out, err := expr.Compile(`name + age`)
// err: invalid operation + (mismatched types string and int)
// | name + age
// | .....^
```
* User-friendly error messages.
* Reasonable set of basic operators.
* Builtins `all`, `none`, `any`, `one`, `filter`, `map`.
```coffeescript
all(Tweets, {.Size <= 280})
```
* Fast ([benchmarks](https://github.com/antonmedv/golang-expression-evaluation-comparison#readme)): uses bytecode virtual machine and optimizing compiler.
## Install
```
go get github.com/antonmedv/expr
```
## Documentation
* See [Getting Started](docs/Getting-Started.md) page for developer documentation.
* See [Language Definition](docs/Language-Definition.md) page to learn the syntax.
## Expr Code Editor
<a href="http://bit.ly/expr-code-editor">
<img src="https://antonmedv.github.io/expr/ogimage.png" align="center" alt="Expr Code Editor" width="1200">
</a>
Also, I have an embeddable code editor written in JavaScript which allows editing expressions with syntax highlighting and autocomplete based on your types declaration.
[Learn more →](https://antonmedv.github.io/expr/)
## Examples
[Play Online](https://play.golang.org/p/z7T8ytJ1T1d)
```go
package main
import (
"fmt"
"github.com/antonmedv/expr"
)
func main() {
env := map[string]interface{}{
"greet": "Hello, %v!",
"names": []string{"world", "you"},
"sprintf": fmt.Sprintf,
}
code := `sprintf(greet, names[0])`
program, err := expr.Compile(code, expr.Env(env))
if err != nil {
panic(err)
}
output, err := expr.Run(program, env)
if err != nil {
panic(err)
}
fmt.Println(output)
}
```
[Play Online](https://play.golang.org/p/4S4brsIvU4i)
```go
package main
import (
"fmt"
"github.com/antonmedv/expr"
)
type Tweet struct {
Len int
}
type Env struct {
Tweets []Tweet
}
func main() {
code := `all(Tweets, {.Len <= 240})`
program, err := expr.Compile(code, expr.Env(Env{}))
if err != nil {
panic(err)
}
env := Env{
Tweets: []Tweet{{42}, {98}, {69}},
}
output, err := expr.Run(program, env)
if err != nil {
panic(err)
}
fmt.Println(output)
}
```
## Contributing
**Expr** consist of a few packages for parsing source code to AST, type checking AST, compiling to bytecode and VM for running bytecode program.
Also expr provides powerful tool [exe](cmd/exe) for debugging. It has interactive terminal debugger for our bytecode virtual machine.
<p align="center">
<img src="docs/images/debug.gif" alt="debugger" width="605">
</p>
## Who is using Expr?
<table>
<tr>
<td>
<p align="center"><a href="https://aviasales.ru"><img alt="Aviasales Logo" height="100" src="https://cdn.worldvectorlogo.com/logos/aviasales-4.svg"></a></p>
<p><a href="https://aviasales.ru">Aviasales</a> are actively using Expr for different parts of the search engine.</p>
</td>
<td>
<p align="center"><a href="https://www.mysteryminds.com/en/"><img alt="Mystery Minds Logo" height="100" src="https://www.mysteryminds.com/wp-content/uploads/2018/07/MM_logo_colours.png"></a></p>
<p><a href="https://www.mysteryminds.com/en/">Mystery Minds</a> uses Expr to allow easy yet powerful customization of its matching algorithm.</p>
</td>
</tr>
</table>
[Add you company too](https://github.com/antonmedv/expr/edit/master/README.md)
## License
[MIT](LICENSE)
+168
View File
@@ -0,0 +1,168 @@
package ast
import (
"reflect"
"regexp"
"github.com/antonmedv/expr/file"
)
// Node represents items of abstract syntax tree.
type Node interface {
Location() file.Location
SetLocation(file.Location)
Type() reflect.Type
SetType(reflect.Type)
}
func Patch(node *Node, newNode Node) {
newNode.SetType((*node).Type())
newNode.SetLocation((*node).Location())
*node = newNode
}
type base struct {
loc file.Location
nodeType reflect.Type
}
func (n *base) Location() file.Location {
return n.loc
}
func (n *base) SetLocation(loc file.Location) {
n.loc = loc
}
func (n *base) Type() reflect.Type {
return n.nodeType
}
func (n *base) SetType(t reflect.Type) {
n.nodeType = t
}
type NilNode struct {
base
}
type IdentifierNode struct {
base
Value string
}
type IntegerNode struct {
base
Value int
}
type FloatNode struct {
base
Value float64
}
type BoolNode struct {
base
Value bool
}
type StringNode struct {
base
Value string
}
type ConstantNode struct {
base
Value interface{}
}
type UnaryNode struct {
base
Operator string
Node Node
}
type BinaryNode struct {
base
Operator string
Left Node
Right Node
}
type MatchesNode struct {
base
Regexp *regexp.Regexp
Left Node
Right Node
}
type PropertyNode struct {
base
Node Node
Property string
}
type IndexNode struct {
base
Node Node
Index Node
}
type SliceNode struct {
base
Node Node
From Node
To Node
}
type MethodNode struct {
base
Node Node
Method string
Arguments []Node
}
type FunctionNode struct {
base
Name string
Arguments []Node
Fast bool
}
type BuiltinNode struct {
base
Name string
Arguments []Node
}
type ClosureNode struct {
base
Node Node
}
type PointerNode struct {
base
}
type ConditionalNode struct {
base
Cond Node
Exp1 Node
Exp2 Node
}
type ArrayNode struct {
base
Nodes []Node
}
type MapNode struct {
base
Pairs []Node
}
type PairNode struct {
base
Key Node
Value Node
}
+59
View File
@@ -0,0 +1,59 @@
package ast
import (
"fmt"
"reflect"
"regexp"
)
func Dump(node Node) string {
return dump(reflect.ValueOf(node), "")
}
func dump(v reflect.Value, ident string) string {
if !v.IsValid() {
return "nil"
}
t := v.Type()
switch t.Kind() {
case reflect.Struct:
out := t.Name() + "{\n"
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
if isPrivate(f.Name) {
continue
}
s := v.Field(i)
out += fmt.Sprintf("%v%v: %v,\n", ident+"\t", f.Name, dump(s, ident+"\t"))
}
return out + ident + "}"
case reflect.Slice:
if v.Len() == 0 {
return "[]"
}
out := "[\n"
for i := 0; i < v.Len(); i++ {
s := v.Index(i)
out += fmt.Sprintf("%v%v,", ident+"\t", dump(s, ident+"\t"))
if i+1 < v.Len() {
out += "\n"
}
}
return out + "\n" + ident + "]"
case reflect.Ptr:
return dump(v.Elem(), ident)
case reflect.Interface:
return dump(reflect.ValueOf(v.Interface()), ident)
case reflect.String:
return fmt.Sprintf("%q", v)
default:
return fmt.Sprintf("%v", v)
}
}
var isCapital = regexp.MustCompile("^[A-Z]")
func isPrivate(s string) bool {
return !isCapital.Match([]byte(s))
}
+108
View File
@@ -0,0 +1,108 @@
package ast
import "fmt"
type Visitor interface {
Enter(node *Node)
Exit(node *Node)
}
type walker struct {
visitor Visitor
}
func Walk(node *Node, visitor Visitor) {
w := walker{
visitor: visitor,
}
w.walk(node)
}
func (w *walker) walk(node *Node) {
w.visitor.Enter(node)
switch n := (*node).(type) {
case *NilNode:
w.visitor.Exit(node)
case *IdentifierNode:
w.visitor.Exit(node)
case *IntegerNode:
w.visitor.Exit(node)
case *FloatNode:
w.visitor.Exit(node)
case *BoolNode:
w.visitor.Exit(node)
case *StringNode:
w.visitor.Exit(node)
case *ConstantNode:
w.visitor.Exit(node)
case *UnaryNode:
w.walk(&n.Node)
w.visitor.Exit(node)
case *BinaryNode:
w.walk(&n.Left)
w.walk(&n.Right)
w.visitor.Exit(node)
case *MatchesNode:
w.walk(&n.Left)
w.walk(&n.Right)
w.visitor.Exit(node)
case *PropertyNode:
w.walk(&n.Node)
w.visitor.Exit(node)
case *IndexNode:
w.walk(&n.Node)
w.walk(&n.Index)
w.visitor.Exit(node)
case *SliceNode:
if n.From != nil {
w.walk(&n.From)
}
if n.To != nil {
w.walk(&n.To)
}
w.visitor.Exit(node)
case *MethodNode:
w.walk(&n.Node)
for i := range n.Arguments {
w.walk(&n.Arguments[i])
}
w.visitor.Exit(node)
case *FunctionNode:
for i := range n.Arguments {
w.walk(&n.Arguments[i])
}
w.visitor.Exit(node)
case *BuiltinNode:
for i := range n.Arguments {
w.walk(&n.Arguments[i])
}
w.visitor.Exit(node)
case *ClosureNode:
w.walk(&n.Node)
w.visitor.Exit(node)
case *PointerNode:
w.visitor.Exit(node)
case *ConditionalNode:
w.walk(&n.Cond)
w.walk(&n.Exp1)
w.walk(&n.Exp2)
w.visitor.Exit(node)
case *ArrayNode:
for i := range n.Nodes {
w.walk(&n.Nodes[i])
}
w.visitor.Exit(node)
case *MapNode:
for i := range n.Pairs {
w.walk(&n.Pairs[i])
}
w.visitor.Exit(node)
case *PairNode:
w.walk(&n.Key)
w.walk(&n.Value)
w.visitor.Exit(node)
default:
panic(fmt.Sprintf("undefined node type (%T)", node))
}
}
+597
View File
@@ -0,0 +1,597 @@
package checker
import (
"fmt"
"reflect"
"github.com/antonmedv/expr/ast"
"github.com/antonmedv/expr/conf"
"github.com/antonmedv/expr/file"
"github.com/antonmedv/expr/parser"
)
func Check(tree *parser.Tree, config *conf.Config) (reflect.Type, error) {
v := &visitor{
collections: make([]reflect.Type, 0),
}
if config != nil {
v.types = config.Types
v.operators = config.Operators
v.expect = config.Expect
v.strict = config.Strict
v.defaultType = config.DefaultType
}
t := v.visit(tree.Node)
if v.expect != reflect.Invalid {
switch v.expect {
case reflect.Int64, reflect.Float64:
if !isNumber(t) {
return nil, fmt.Errorf("expected %v, but got %v", v.expect, t)
}
default:
if t.Kind() != v.expect {
return nil, fmt.Errorf("expected %v, but got %v", v.expect, t)
}
}
}
if v.err != nil {
return t, v.err.Bind(tree.Source)
}
return t, nil
}
type visitor struct {
types conf.TypesTable
operators conf.OperatorsTable
expect reflect.Kind
collections []reflect.Type
strict bool
defaultType reflect.Type
err *file.Error
}
func (v *visitor) visit(node ast.Node) reflect.Type {
var t reflect.Type
switch n := node.(type) {
case *ast.NilNode:
t = v.NilNode(n)
case *ast.IdentifierNode:
t = v.IdentifierNode(n)
case *ast.IntegerNode:
t = v.IntegerNode(n)
case *ast.FloatNode:
t = v.FloatNode(n)
case *ast.BoolNode:
t = v.BoolNode(n)
case *ast.StringNode:
t = v.StringNode(n)
case *ast.UnaryNode:
t = v.UnaryNode(n)
case *ast.BinaryNode:
t = v.BinaryNode(n)
case *ast.MatchesNode:
t = v.MatchesNode(n)
case *ast.PropertyNode:
t = v.PropertyNode(n)
case *ast.IndexNode:
t = v.IndexNode(n)
case *ast.SliceNode:
t = v.SliceNode(n)
case *ast.MethodNode:
t = v.MethodNode(n)
case *ast.FunctionNode:
t = v.FunctionNode(n)
case *ast.BuiltinNode:
t = v.BuiltinNode(n)
case *ast.ClosureNode:
t = v.ClosureNode(n)
case *ast.PointerNode:
t = v.PointerNode(n)
case *ast.ConditionalNode:
t = v.ConditionalNode(n)
case *ast.ArrayNode:
t = v.ArrayNode(n)
case *ast.MapNode:
t = v.MapNode(n)
case *ast.PairNode:
t = v.PairNode(n)
default:
panic(fmt.Sprintf("undefined node type (%T)", node))
}
node.SetType(t)
return t
}
func (v *visitor) error(node ast.Node, format string, args ...interface{}) reflect.Type {
if v.err == nil { // show first error
v.err = &file.Error{
Location: node.Location(),
Message: fmt.Sprintf(format, args...),
}
}
return interfaceType // interface represent undefined type
}
func (v *visitor) NilNode(*ast.NilNode) reflect.Type {
return nilType
}
func (v *visitor) IdentifierNode(node *ast.IdentifierNode) reflect.Type {
if v.types == nil {
return interfaceType
}
if t, ok := v.types[node.Value]; ok {
if t.Ambiguous {
return v.error(node, "ambiguous identifier %v", node.Value)
}
return t.Type
}
if !v.strict {
if v.defaultType != nil {
return v.defaultType
}
return interfaceType
}
return v.error(node, "unknown name %v", node.Value)
}
func (v *visitor) IntegerNode(*ast.IntegerNode) reflect.Type {
return integerType
}
func (v *visitor) FloatNode(*ast.FloatNode) reflect.Type {
return floatType
}
func (v *visitor) BoolNode(*ast.BoolNode) reflect.Type {
return boolType
}
func (v *visitor) StringNode(*ast.StringNode) reflect.Type {
return stringType
}
func (v *visitor) UnaryNode(node *ast.UnaryNode) reflect.Type {
t := v.visit(node.Node)
switch node.Operator {
case "!", "not":
if isBool(t) {
return boolType
}
case "+", "-":
if isNumber(t) {
return t
}
default:
return v.error(node, "unknown operator (%v)", node.Operator)
}
return v.error(node, `invalid operation: %v (mismatched type %v)`, node.Operator, t)
}
func (v *visitor) BinaryNode(node *ast.BinaryNode) reflect.Type {
l := v.visit(node.Left)
r := v.visit(node.Right)
// check operator overloading
if fns, ok := v.operators[node.Operator]; ok {
t, _, ok := conf.FindSuitableOperatorOverload(fns, v.types, l, r)
if ok {
return t
}
}
switch node.Operator {
case "==", "!=":
if isNumber(l) && isNumber(r) {
return boolType
}
if isComparable(l, r) {
return boolType
}
case "or", "||", "and", "&&":
if isBool(l) && isBool(r) {
return boolType
}
case "in", "not in":
if isString(l) && isStruct(r) {
return boolType
}
if isMap(r) {
return boolType
}
if isArray(r) {
return boolType
}
case "<", ">", ">=", "<=":
if isNumber(l) && isNumber(r) {
return boolType
}
if isString(l) && isString(r) {
return boolType
}
case "/", "-", "*":
if isNumber(l) && isNumber(r) {
return combined(l, r)
}
case "**":
if isNumber(l) && isNumber(r) {
return floatType
}
case "%":
if isInteger(l) && isInteger(r) {
return combined(l, r)
}
case "+":
if isNumber(l) && isNumber(r) {
return combined(l, r)
}
if isString(l) && isString(r) {
return stringType
}
case "contains", "startsWith", "endsWith":
if isString(l) && isString(r) {
return boolType
}
case "..":
if isInteger(l) && isInteger(r) {
return reflect.SliceOf(integerType)
}
default:
return v.error(node, "unknown operator (%v)", node.Operator)
}
return v.error(node, `invalid operation: %v (mismatched types %v and %v)`, node.Operator, l, r)
}
func (v *visitor) MatchesNode(node *ast.MatchesNode) reflect.Type {
l := v.visit(node.Left)
r := v.visit(node.Right)
if isString(l) && isString(r) {
return boolType
}
return v.error(node, `invalid operation: matches (mismatched types %v and %v)`, l, r)
}
func (v *visitor) PropertyNode(node *ast.PropertyNode) reflect.Type {
t := v.visit(node.Node)
if t, ok := fieldType(t, node.Property); ok {
return t
}
return v.error(node, "type %v has no field %v", t, node.Property)
}
func (v *visitor) IndexNode(node *ast.IndexNode) reflect.Type {
t := v.visit(node.Node)
i := v.visit(node.Index)
if t, ok := indexType(t); ok {
if !isInteger(i) && !isString(i) {
return v.error(node, "invalid operation: cannot use %v as index to %v", i, t)
}
return t
}
return v.error(node, "invalid operation: type %v does not support indexing", t)
}
func (v *visitor) SliceNode(node *ast.SliceNode) reflect.Type {
t := v.visit(node.Node)
_, isIndex := indexType(t)
if isIndex || isString(t) {
if node.From != nil {
from := v.visit(node.From)
if !isInteger(from) {
return v.error(node.From, "invalid operation: non-integer slice index %v", from)
}
}
if node.To != nil {
to := v.visit(node.To)
if !isInteger(to) {
return v.error(node.To, "invalid operation: non-integer slice index %v", to)
}
}
return t
}
return v.error(node, "invalid operation: cannot slice %v", t)
}
func (v *visitor) FunctionNode(node *ast.FunctionNode) reflect.Type {
if f, ok := v.types[node.Name]; ok {
if fn, ok := isFuncType(f.Type); ok {
inputParamsCount := 1 // for functions
if f.Method {
inputParamsCount = 2 // for methods
}
if !isInterface(fn) &&
fn.IsVariadic() &&
fn.NumIn() == inputParamsCount &&
fn.NumOut() == 1 &&
fn.Out(0).Kind() == reflect.Interface {
rest := fn.In(fn.NumIn() - 1) // function has only one param for functions and two for methods
if rest.Kind() == reflect.Slice && rest.Elem().Kind() == reflect.Interface {
node.Fast = true
}
}
return v.checkFunc(fn, f.Method, node, node.Name, node.Arguments)
}
}
if !v.strict {
if v.defaultType != nil {
return v.defaultType
}
return interfaceType
}
return v.error(node, "unknown func %v", node.Name)
}
func (v *visitor) MethodNode(node *ast.MethodNode) reflect.Type {
t := v.visit(node.Node)
if f, method, ok := methodType(t, node.Method); ok {
if fn, ok := isFuncType(f); ok {
return v.checkFunc(fn, method, node, node.Method, node.Arguments)
}
}
return v.error(node, "type %v has no method %v", t, node.Method)
}
// checkFunc checks func arguments and returns "return type" of func or method.
func (v *visitor) checkFunc(fn reflect.Type, method bool, node ast.Node, name string, arguments []ast.Node) reflect.Type {
if isInterface(fn) {
return interfaceType
}
if fn.NumOut() == 0 {
return v.error(node, "func %v doesn't return value", name)
}
if fn.NumOut() != 1 {
return v.error(node, "func %v returns more then one value", name)
}
numIn := fn.NumIn()
// If func is method on an env, first argument should be a receiver,
// and actual arguments less then numIn by one.
if method {
numIn--
}
if fn.IsVariadic() {
if len(arguments) < numIn-1 {
return v.error(node, "not enough arguments to call %v", name)
}
} else {
if len(arguments) > numIn {
return v.error(node, "too many arguments to call %v", name)
}
if len(arguments) < numIn {
return v.error(node, "not enough arguments to call %v", name)
}
}
offset := 0
// Skip first argument in case of the receiver.
if method {
offset = 1
}
for i, arg := range arguments {
t := v.visit(arg)
var in reflect.Type
if fn.IsVariadic() && i >= numIn-1 {
// For variadic arguments fn(xs ...int), go replaces type of xs (int) with ([]int).
// As we compare arguments one by one, we need underling type.
in = fn.In(fn.NumIn() - 1)
in, _ = indexType(in)
} else {
in = fn.In(i + offset)
}
if isIntegerOrArithmeticOperation(arg) {
t = in
setTypeForIntegers(arg, t)
}
if t == nil {
continue
}
if !t.AssignableTo(in) && t.Kind() != reflect.Interface {
return v.error(arg, "cannot use %v as argument (type %v) to call %v ", t, in, name)
}
}
return fn.Out(0)
}
func (v *visitor) BuiltinNode(node *ast.BuiltinNode) reflect.Type {
switch node.Name {
case "len":
param := v.visit(node.Arguments[0])
if isArray(param) || isMap(param) || isString(param) {
return integerType
}
return v.error(node, "invalid argument for len (type %v)", param)
case "all", "none", "any", "one":
collection := v.visit(node.Arguments[0])
if !isArray(collection) {
return v.error(node.Arguments[0], "builtin %v takes only array (got %v)", node.Name, collection)
}
v.collections = append(v.collections, collection)
closure := v.visit(node.Arguments[1])
v.collections = v.collections[:len(v.collections)-1]
if isFunc(closure) &&
closure.NumOut() == 1 &&
closure.NumIn() == 1 && isInterface(closure.In(0)) {
if !isBool(closure.Out(0)) {
return v.error(node.Arguments[1], "closure should return boolean (got %v)", closure.Out(0).String())
}
return boolType
}
return v.error(node.Arguments[1], "closure should has one input and one output param")
case "filter":
collection := v.visit(node.Arguments[0])
if !isArray(collection) {
return v.error(node.Arguments[0], "builtin %v takes only array (got %v)", node.Name, collection)
}
v.collections = append(v.collections, collection)
closure := v.visit(node.Arguments[1])
v.collections = v.collections[:len(v.collections)-1]
if isFunc(closure) &&
closure.NumOut() == 1 &&
closure.NumIn() == 1 && isInterface(closure.In(0)) {
if !isBool(closure.Out(0)) {
return v.error(node.Arguments[1], "closure should return boolean (got %v)", closure.Out(0).String())
}
if isInterface(collection) {
return arrayType
}
return reflect.SliceOf(collection.Elem())
}
return v.error(node.Arguments[1], "closure should has one input and one output param")
case "map":
collection := v.visit(node.Arguments[0])
if !isArray(collection) {
return v.error(node.Arguments[0], "builtin %v takes only array (got %v)", node.Name, collection)
}
v.collections = append(v.collections, collection)
closure := v.visit(node.Arguments[1])
v.collections = v.collections[:len(v.collections)-1]
if isFunc(closure) &&
closure.NumOut() == 1 &&
closure.NumIn() == 1 && isInterface(closure.In(0)) {
return reflect.SliceOf(closure.Out(0))
}
return v.error(node.Arguments[1], "closure should has one input and one output param")
case "count":
collection := v.visit(node.Arguments[0])
if !isArray(collection) {
return v.error(node.Arguments[0], "builtin %v takes only array (got %v)", node.Name, collection)
}
v.collections = append(v.collections, collection)
closure := v.visit(node.Arguments[1])
v.collections = v.collections[:len(v.collections)-1]
if isFunc(closure) &&
closure.NumOut() == 1 &&
closure.NumIn() == 1 && isInterface(closure.In(0)) {
if !isBool(closure.Out(0)) {
return v.error(node.Arguments[1], "closure should return boolean (got %v)", closure.Out(0).String())
}
return integerType
}
return v.error(node.Arguments[1], "closure should has one input and one output param")
default:
return v.error(node, "unknown builtin %v", node.Name)
}
}
func (v *visitor) ClosureNode(node *ast.ClosureNode) reflect.Type {
t := v.visit(node.Node)
return reflect.FuncOf([]reflect.Type{interfaceType}, []reflect.Type{t}, false)
}
func (v *visitor) PointerNode(node *ast.PointerNode) reflect.Type {
if len(v.collections) == 0 {
return v.error(node, "cannot use pointer accessor outside closure")
}
collection := v.collections[len(v.collections)-1]
if t, ok := indexType(collection); ok {
return t
}
return v.error(node, "cannot use %v as array", collection)
}
func (v *visitor) ConditionalNode(node *ast.ConditionalNode) reflect.Type {
c := v.visit(node.Cond)
if !isBool(c) {
return v.error(node.Cond, "non-bool expression (type %v) used as condition", c)
}
t1 := v.visit(node.Exp1)
t2 := v.visit(node.Exp2)
if t1 == nil && t2 != nil {
return t2
}
if t1 != nil && t2 == nil {
return t1
}
if t1 == nil && t2 == nil {
return nilType
}
if t1.AssignableTo(t2) {
return t1
}
return interfaceType
}
func (v *visitor) ArrayNode(node *ast.ArrayNode) reflect.Type {
for _, node := range node.Nodes {
_ = v.visit(node)
}
return arrayType
}
func (v *visitor) MapNode(node *ast.MapNode) reflect.Type {
for _, pair := range node.Pairs {
v.visit(pair)
}
return mapType
}
func (v *visitor) PairNode(node *ast.PairNode) reflect.Type {
v.visit(node.Key)
v.visit(node.Value)
return nilType
}
+349
View File
@@ -0,0 +1,349 @@
package checker
import (
"reflect"
"github.com/antonmedv/expr/ast"
)
var (
nilType = reflect.TypeOf(nil)
boolType = reflect.TypeOf(true)
integerType = reflect.TypeOf(int(0))
floatType = reflect.TypeOf(float64(0))
stringType = reflect.TypeOf("")
arrayType = reflect.TypeOf([]interface{}{})
mapType = reflect.TypeOf(map[string]interface{}{})
interfaceType = reflect.TypeOf(new(interface{})).Elem()
)
func typeWeight(t reflect.Type) int {
switch t.Kind() {
case reflect.Uint:
return 1
case reflect.Uint8:
return 2
case reflect.Uint16:
return 3
case reflect.Uint32:
return 4
case reflect.Uint64:
return 5
case reflect.Int:
return 6
case reflect.Int8:
return 7
case reflect.Int16:
return 8
case reflect.Int32:
return 9
case reflect.Int64:
return 10
case reflect.Float32:
return 11
case reflect.Float64:
return 12
default:
return 0
}
}
func combined(a, b reflect.Type) reflect.Type {
if typeWeight(a) > typeWeight(b) {
return a
} else {
return b
}
}
func dereference(t reflect.Type) reflect.Type {
if t == nil {
return nil
}
if t.Kind() == reflect.Ptr {
t = dereference(t.Elem())
}
return t
}
func isComparable(l, r reflect.Type) bool {
l = dereference(l)
r = dereference(r)
if l == nil || r == nil { // It is possible to compare with nil.
return true
}
if l.Kind() == r.Kind() {
return true
}
if isInterface(l) || isInterface(r) {
return true
}
return false
}
func isInterface(t reflect.Type) bool {
t = dereference(t)
if t != nil {
switch t.Kind() {
case reflect.Interface:
return true
}
}
return false
}
func isInteger(t reflect.Type) bool {
t = dereference(t)
if t != nil {
switch t.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
fallthrough
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
return true
case reflect.Interface:
return true
}
}
return false
}
func isFloat(t reflect.Type) bool {
t = dereference(t)
if t != nil {
switch t.Kind() {
case reflect.Float32, reflect.Float64:
return true
case reflect.Interface:
return true
}
}
return false
}
func isNumber(t reflect.Type) bool {
return isInteger(t) || isFloat(t)
}
func isBool(t reflect.Type) bool {
t = dereference(t)
if t != nil {
switch t.Kind() {
case reflect.Bool:
return true
case reflect.Interface:
return true
}
}
return false
}
func isString(t reflect.Type) bool {
t = dereference(t)
if t != nil {
switch t.Kind() {
case reflect.String:
return true
case reflect.Interface:
return true
}
}
return false
}
func isArray(t reflect.Type) bool {
t = dereference(t)
if t != nil {
switch t.Kind() {
case reflect.Slice, reflect.Array:
return true
case reflect.Interface:
return true
}
}
return false
}
func isMap(t reflect.Type) bool {
t = dereference(t)
if t != nil {
switch t.Kind() {
case reflect.Map:
return true
case reflect.Interface:
return true
}
}
return false
}
func isStruct(t reflect.Type) bool {
t = dereference(t)
if t != nil {
switch t.Kind() {
case reflect.Struct:
return true
}
}
return false
}
func isFunc(t reflect.Type) bool {
t = dereference(t)
if t != nil {
switch t.Kind() {
case reflect.Func:
return true
}
}
return false
}
func fieldType(ntype reflect.Type, name string) (reflect.Type, bool) {
ntype = dereference(ntype)
if ntype != nil {
switch ntype.Kind() {
case reflect.Interface:
return interfaceType, true
case reflect.Struct:
// First check all struct's fields.
for i := 0; i < ntype.NumField(); i++ {
f := ntype.Field(i)
if f.Name == name {
return f.Type, true
}
}
// Second check fields of embedded structs.
for i := 0; i < ntype.NumField(); i++ {
f := ntype.Field(i)
if f.Anonymous {
if t, ok := fieldType(f.Type, name); ok {
return t, true
}
}
}
case reflect.Map:
return ntype.Elem(), true
}
}
return nil, false
}
func methodType(t reflect.Type, name string) (reflect.Type, bool, bool) {
if t != nil {
// First, check methods defined on type itself,
// independent of which type it is.
if m, ok := t.MethodByName(name); ok {
if t.Kind() == reflect.Interface {
// In case of interface type method will not have a receiver,
// and to prevent checker decreasing numbers of in arguments
// return method type as not method (second argument is false).
return m.Type, false, true
} else {
return m.Type, true, true
}
}
d := t
if t.Kind() == reflect.Ptr {
d = t.Elem()
}
switch d.Kind() {
case reflect.Interface:
return interfaceType, false, true
case reflect.Struct:
// First, check all struct's fields.
for i := 0; i < d.NumField(); i++ {
f := d.Field(i)
if !f.Anonymous && f.Name == name {
return f.Type, false, true
}
}
// Second, check fields of embedded structs.
for i := 0; i < d.NumField(); i++ {
f := d.Field(i)
if f.Anonymous {
if t, method, ok := methodType(f.Type, name); ok {
return t, method, true
}
}
}
case reflect.Map:
return d.Elem(), false, true
}
}
return nil, false, false
}
func indexType(ntype reflect.Type) (reflect.Type, bool) {
ntype = dereference(ntype)
if ntype == nil {
return nil, false
}
switch ntype.Kind() {
case reflect.Interface:
return interfaceType, true
case reflect.Map, reflect.Array, reflect.Slice:
return ntype.Elem(), true
}
return nil, false
}
func isFuncType(ntype reflect.Type) (reflect.Type, bool) {
ntype = dereference(ntype)
if ntype == nil {
return nil, false
}
switch ntype.Kind() {
case reflect.Interface:
return interfaceType, true
case reflect.Func:
return ntype, true
}
return nil, false
}
func isIntegerOrArithmeticOperation(node ast.Node) bool {
switch n := node.(type) {
case *ast.IntegerNode:
return true
case *ast.UnaryNode:
switch n.Operator {
case "+", "-":
return true
}
case *ast.BinaryNode:
switch n.Operator {
case "+", "/", "-", "*":
return true
}
}
return false
}
func setTypeForIntegers(node ast.Node, t reflect.Type) {
switch n := node.(type) {
case *ast.IntegerNode:
n.SetType(t)
case *ast.UnaryNode:
switch n.Operator {
case "+", "-":
setTypeForIntegers(n.Node, t)
}
case *ast.BinaryNode:
switch n.Operator {
case "+", "/", "-", "*":
setTypeForIntegers(n.Left, t)
setTypeForIntegers(n.Right, t)
}
}
}
+663
View File
@@ -0,0 +1,663 @@
package compiler
import (
"encoding/binary"
"fmt"
"math"
"reflect"
"github.com/antonmedv/expr/ast"
"github.com/antonmedv/expr/conf"
"github.com/antonmedv/expr/file"
"github.com/antonmedv/expr/parser"
. "github.com/antonmedv/expr/vm"
)
func Compile(tree *parser.Tree, config *conf.Config) (program *Program, err error) {
defer func() {
if r := recover(); r != nil {
err = fmt.Errorf("%v", r)
}
}()
c := &compiler{
index: make(map[interface{}]uint16),
locations: make(map[int]file.Location),
}
if config != nil {
c.mapEnv = config.MapEnv
c.cast = config.Expect
}
c.compile(tree.Node)
switch c.cast {
case reflect.Int64:
c.emit(OpCast, encode(0)...)
case reflect.Float64:
c.emit(OpCast, encode(1)...)
}
program = &Program{
Source: tree.Source,
Locations: c.locations,
Constants: c.constants,
Bytecode: c.bytecode,
}
return
}
type compiler struct {
locations map[int]file.Location
constants []interface{}
bytecode []byte
index map[interface{}]uint16
mapEnv bool
cast reflect.Kind
nodes []ast.Node
}
func (c *compiler) emit(op byte, b ...byte) int {
c.bytecode = append(c.bytecode, op)
current := len(c.bytecode)
c.bytecode = append(c.bytecode, b...)
var loc file.Location
if len(c.nodes) > 0 {
loc = c.nodes[len(c.nodes)-1].Location()
}
c.locations[current-1] = loc
return current
}
func (c *compiler) emitPush(value interface{}) int {
return c.emit(OpPush, c.makeConstant(value)...)
}
func (c *compiler) makeConstant(i interface{}) []byte {
hashable := true
switch reflect.TypeOf(i).Kind() {
case reflect.Slice, reflect.Map:
hashable = false
}
if hashable {
if p, ok := c.index[i]; ok {
return encode(p)
}
}
c.constants = append(c.constants, i)
if len(c.constants) > math.MaxUint16 {
panic("exceeded constants max space limit")
}
p := uint16(len(c.constants) - 1)
if hashable {
c.index[i] = p
}
return encode(p)
}
func (c *compiler) placeholder() []byte {
return []byte{0xFF, 0xFF}
}
func (c *compiler) patchJump(placeholder int) {
offset := len(c.bytecode) - 2 - placeholder
b := encode(uint16(offset))
c.bytecode[placeholder] = b[0]
c.bytecode[placeholder+1] = b[1]
}
func (c *compiler) calcBackwardJump(to int) []byte {
return encode(uint16(len(c.bytecode) + 1 + 2 - to))
}
func (c *compiler) compile(node ast.Node) {
c.nodes = append(c.nodes, node)
defer func() {
c.nodes = c.nodes[:len(c.nodes)-1]
}()
switch n := node.(type) {
case *ast.NilNode:
c.NilNode(n)
case *ast.IdentifierNode:
c.IdentifierNode(n)
case *ast.IntegerNode:
c.IntegerNode(n)
case *ast.FloatNode:
c.FloatNode(n)
case *ast.BoolNode:
c.BoolNode(n)
case *ast.StringNode:
c.StringNode(n)
case *ast.ConstantNode:
c.ConstantNode(n)
case *ast.UnaryNode:
c.UnaryNode(n)
case *ast.BinaryNode:
c.BinaryNode(n)
case *ast.MatchesNode:
c.MatchesNode(n)
case *ast.PropertyNode:
c.PropertyNode(n)
case *ast.IndexNode:
c.IndexNode(n)
case *ast.SliceNode:
c.SliceNode(n)
case *ast.MethodNode:
c.MethodNode(n)
case *ast.FunctionNode:
c.FunctionNode(n)
case *ast.BuiltinNode:
c.BuiltinNode(n)
case *ast.ClosureNode:
c.ClosureNode(n)
case *ast.PointerNode:
c.PointerNode(n)
case *ast.ConditionalNode:
c.ConditionalNode(n)
case *ast.ArrayNode:
c.ArrayNode(n)
case *ast.MapNode:
c.MapNode(n)
case *ast.PairNode:
c.PairNode(n)
default:
panic(fmt.Sprintf("undefined node type (%T)", node))
}
}
func (c *compiler) NilNode(node *ast.NilNode) {
c.emit(OpNil)
}
func (c *compiler) IdentifierNode(node *ast.IdentifierNode) {
v := c.makeConstant(node.Value)
if c.mapEnv {
c.emit(OpFetchMap, v...)
} else {
c.emit(OpFetch, v...)
}
}
func (c *compiler) IntegerNode(node *ast.IntegerNode) {
t := node.Type()
if t == nil {
c.emitPush(node.Value)
return
}
switch t.Kind() {
case reflect.Float32:
c.emitPush(float32(node.Value))
case reflect.Float64:
c.emitPush(float64(node.Value))
case reflect.Int:
c.emitPush(int(node.Value))
case reflect.Int8:
c.emitPush(int8(node.Value))
case reflect.Int16:
c.emitPush(int16(node.Value))
case reflect.Int32:
c.emitPush(int32(node.Value))
case reflect.Int64:
c.emitPush(int64(node.Value))
case reflect.Uint:
c.emitPush(uint(node.Value))
case reflect.Uint8:
c.emitPush(uint8(node.Value))
case reflect.Uint16:
c.emitPush(uint16(node.Value))
case reflect.Uint32:
c.emitPush(uint32(node.Value))
case reflect.Uint64:
c.emitPush(uint64(node.Value))
default:
c.emitPush(node.Value)
}
}
func (c *compiler) FloatNode(node *ast.FloatNode) {
c.emitPush(node.Value)
}
func (c *compiler) BoolNode(node *ast.BoolNode) {
if node.Value {
c.emit(OpTrue)
} else {
c.emit(OpFalse)
}
}
func (c *compiler) StringNode(node *ast.StringNode) {
c.emitPush(node.Value)
}
func (c *compiler) ConstantNode(node *ast.ConstantNode) {
c.emitPush(node.Value)
}
func (c *compiler) UnaryNode(node *ast.UnaryNode) {
c.compile(node.Node)
switch node.Operator {
case "!", "not":
c.emit(OpNot)
case "+":
// Do nothing
case "-":
c.emit(OpNegate)
default:
panic(fmt.Sprintf("unknown operator (%v)", node.Operator))
}
}
func (c *compiler) BinaryNode(node *ast.BinaryNode) {
l := kind(node.Left)
r := kind(node.Right)
switch node.Operator {
case "==":
c.compile(node.Left)
c.compile(node.Right)
if l == r && l == reflect.Int {
c.emit(OpEqualInt)
} else if l == r && l == reflect.String {
c.emit(OpEqualString)
} else {
c.emit(OpEqual)
}
case "!=":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpEqual)
c.emit(OpNot)
case "or", "||":
c.compile(node.Left)
end := c.emit(OpJumpIfTrue, c.placeholder()...)
c.emit(OpPop)
c.compile(node.Right)
c.patchJump(end)
case "and", "&&":
c.compile(node.Left)
end := c.emit(OpJumpIfFalse, c.placeholder()...)
c.emit(OpPop)
c.compile(node.Right)
c.patchJump(end)
case "in":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpIn)
case "not in":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpIn)
c.emit(OpNot)
case "<":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpLess)
case ">":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpMore)
case "<=":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpLessOrEqual)
case ">=":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpMoreOrEqual)
case "+":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpAdd)
case "-":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpSubtract)
case "*":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpMultiply)
case "/":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpDivide)
case "%":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpModulo)
case "**":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpExponent)
case "contains":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpContains)
case "startsWith":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpStartsWith)
case "endsWith":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpEndsWith)
case "..":
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpRange)
default:
panic(fmt.Sprintf("unknown operator (%v)", node.Operator))
}
}
func (c *compiler) MatchesNode(node *ast.MatchesNode) {
if node.Regexp != nil {
c.compile(node.Left)
c.emit(OpMatchesConst, c.makeConstant(node.Regexp)...)
return
}
c.compile(node.Left)
c.compile(node.Right)
c.emit(OpMatches)
}
func (c *compiler) PropertyNode(node *ast.PropertyNode) {
c.compile(node.Node)
c.emit(OpProperty, c.makeConstant(node.Property)...)
}
func (c *compiler) IndexNode(node *ast.IndexNode) {
c.compile(node.Node)
c.compile(node.Index)
c.emit(OpIndex)
}
func (c *compiler) SliceNode(node *ast.SliceNode) {
c.compile(node.Node)
if node.To != nil {
c.compile(node.To)
} else {
c.emit(OpLen)
}
if node.From != nil {
c.compile(node.From)
} else {
c.emitPush(0)
}
c.emit(OpSlice)
}
func (c *compiler) MethodNode(node *ast.MethodNode) {
c.compile(node.Node)
for _, arg := range node.Arguments {
c.compile(arg)
}
c.emit(OpMethod, c.makeConstant(Call{Name: node.Method, Size: len(node.Arguments)})...)
}
func (c *compiler) FunctionNode(node *ast.FunctionNode) {
for _, arg := range node.Arguments {
c.compile(arg)
}
op := OpCall
if node.Fast {
op = OpCallFast
}
c.emit(op, c.makeConstant(Call{Name: node.Name, Size: len(node.Arguments)})...)
}
func (c *compiler) BuiltinNode(node *ast.BuiltinNode) {
switch node.Name {
case "len":
c.compile(node.Arguments[0])
c.emit(OpLen)
c.emit(OpRot)
c.emit(OpPop)
case "all":
c.compile(node.Arguments[0])
c.emit(OpBegin)
var loopBreak int
c.emitLoop(func() {
c.compile(node.Arguments[1])
loopBreak = c.emit(OpJumpIfFalse, c.placeholder()...)
c.emit(OpPop)
})
c.emit(OpTrue)
c.patchJump(loopBreak)
c.emit(OpEnd)
case "none":
c.compile(node.Arguments[0])
c.emit(OpBegin)
var loopBreak int
c.emitLoop(func() {
c.compile(node.Arguments[1])
c.emit(OpNot)
loopBreak = c.emit(OpJumpIfFalse, c.placeholder()...)
c.emit(OpPop)
})
c.emit(OpTrue)
c.patchJump(loopBreak)
c.emit(OpEnd)
case "any":
c.compile(node.Arguments[0])
c.emit(OpBegin)
var loopBreak int
c.emitLoop(func() {
c.compile(node.Arguments[1])
loopBreak = c.emit(OpJumpIfTrue, c.placeholder()...)
c.emit(OpPop)
})
c.emit(OpFalse)
c.patchJump(loopBreak)
c.emit(OpEnd)
case "one":
count := c.makeConstant("count")
c.compile(node.Arguments[0])
c.emit(OpBegin)
c.emitPush(0)
c.emit(OpStore, count...)
c.emitLoop(func() {
c.compile(node.Arguments[1])
c.emitCond(func() {
c.emit(OpInc, count...)
})
})
c.emit(OpLoad, count...)
c.emitPush(1)
c.emit(OpEqual)
c.emit(OpEnd)
case "filter":
count := c.makeConstant("count")
c.compile(node.Arguments[0])
c.emit(OpBegin)
c.emitPush(0)
c.emit(OpStore, count...)
c.emitLoop(func() {
c.compile(node.Arguments[1])
c.emitCond(func() {
c.emit(OpInc, count...)
c.emit(OpLoad, c.makeConstant("array")...)
c.emit(OpLoad, c.makeConstant("i")...)
c.emit(OpIndex)
})
})
c.emit(OpLoad, count...)
c.emit(OpEnd)
c.emit(OpArray)
case "map":
c.compile(node.Arguments[0])
c.emit(OpBegin)
size := c.emitLoop(func() {
c.compile(node.Arguments[1])
})
c.emit(OpLoad, size...)
c.emit(OpEnd)
c.emit(OpArray)
case "count":
count := c.makeConstant("count")
c.compile(node.Arguments[0])
c.emit(OpBegin)
c.emitPush(0)
c.emit(OpStore, count...)
c.emitLoop(func() {
c.compile(node.Arguments[1])
c.emitCond(func() {
c.emit(OpInc, count...)
})
})
c.emit(OpLoad, count...)
c.emit(OpEnd)
default:
panic(fmt.Sprintf("unknown builtin %v", node.Name))
}
}
func (c *compiler) emitCond(body func()) {
noop := c.emit(OpJumpIfFalse, c.placeholder()...)
c.emit(OpPop)
body()
jmp := c.emit(OpJump, c.placeholder()...)
c.patchJump(noop)
c.emit(OpPop)
c.patchJump(jmp)
}
func (c *compiler) emitLoop(body func()) []byte {
i := c.makeConstant("i")
size := c.makeConstant("size")
array := c.makeConstant("array")
c.emit(OpLen)
c.emit(OpStore, size...)
c.emit(OpStore, array...)
c.emitPush(0)
c.emit(OpStore, i...)
cond := len(c.bytecode)
c.emit(OpLoad, i...)
c.emit(OpLoad, size...)
c.emit(OpLess)
end := c.emit(OpJumpIfFalse, c.placeholder()...)
c.emit(OpPop)
body()
c.emit(OpInc, i...)
c.emit(OpJumpBackward, c.calcBackwardJump(cond)...)
c.patchJump(end)
c.emit(OpPop)
return size
}
func (c *compiler) ClosureNode(node *ast.ClosureNode) {
c.compile(node.Node)
}
func (c *compiler) PointerNode(node *ast.PointerNode) {
c.emit(OpLoad, c.makeConstant("array")...)
c.emit(OpLoad, c.makeConstant("i")...)
c.emit(OpIndex)
}
func (c *compiler) ConditionalNode(node *ast.ConditionalNode) {
c.compile(node.Cond)
otherwise := c.emit(OpJumpIfFalse, c.placeholder()...)
c.emit(OpPop)
c.compile(node.Exp1)
end := c.emit(OpJump, c.placeholder()...)
c.patchJump(otherwise)
c.emit(OpPop)
c.compile(node.Exp2)
c.patchJump(end)
}
func (c *compiler) ArrayNode(node *ast.ArrayNode) {
for _, node := range node.Nodes {
c.compile(node)
}
c.emitPush(len(node.Nodes))
c.emit(OpArray)
}
func (c *compiler) MapNode(node *ast.MapNode) {
for _, pair := range node.Pairs {
c.compile(pair)
}
c.emitPush(len(node.Pairs))
c.emit(OpMap)
}
func (c *compiler) PairNode(node *ast.PairNode) {
c.compile(node.Key)
c.compile(node.Value)
}
func encode(i uint16) []byte {
b := make([]byte, 2)
binary.LittleEndian.PutUint16(b, i)
return b
}
func kind(node ast.Node) reflect.Kind {
t := node.Type()
if t == nil {
return reflect.Invalid
}
return t.Kind()
}
+44
View File
@@ -0,0 +1,44 @@
package compiler
import (
"github.com/antonmedv/expr/ast"
"github.com/antonmedv/expr/conf"
)
type operatorPatcher struct {
ops map[string][]string
types conf.TypesTable
}
func (p *operatorPatcher) Enter(node *ast.Node) {}
func (p *operatorPatcher) Exit(node *ast.Node) {
binaryNode, ok := (*node).(*ast.BinaryNode)
if !ok {
return
}
fns, ok := p.ops[binaryNode.Operator]
if !ok {
return
}
leftType := binaryNode.Left.Type()
rightType := binaryNode.Right.Type()
_, fn, ok := conf.FindSuitableOperatorOverload(fns, p.types, leftType, rightType)
if ok {
newNode := &ast.FunctionNode{
Name: fn,
Arguments: []ast.Node{binaryNode.Left, binaryNode.Right},
}
ast.Patch(node, newNode)
}
}
func PatchOperators(node *ast.Node, config *conf.Config) {
if len(config.Operators) == 0 {
return
}
patcher := &operatorPatcher{ops: config.Operators, types: config.Types}
ast.Walk(node, patcher)
}
+89
View File
@@ -0,0 +1,89 @@
package conf
import (
"fmt"
"reflect"
"github.com/antonmedv/expr/ast"
"github.com/antonmedv/expr/vm"
)
type Config struct {
Env interface{}
MapEnv bool
Types TypesTable
Operators OperatorsTable
Expect reflect.Kind
Optimize bool
Strict bool
DefaultType reflect.Type
ConstExprFns map[string]reflect.Value
Visitors []ast.Visitor
err error
}
func New(env interface{}) *Config {
var mapEnv bool
var mapValueType reflect.Type
if _, ok := env.(map[string]interface{}); ok {
mapEnv = true
} else {
if reflect.ValueOf(env).Kind() == reflect.Map {
mapValueType = reflect.TypeOf(env).Elem()
}
}
return &Config{
Env: env,
MapEnv: mapEnv,
Types: CreateTypesTable(env),
Optimize: true,
Strict: true,
DefaultType: mapValueType,
ConstExprFns: make(map[string]reflect.Value),
}
}
// Check validates the compiler configuration.
func (c *Config) Check() error {
// Check that all functions that define operator overloading
// exist in environment and have correct signatures.
for op, fns := range c.Operators {
for _, fn := range fns {
fnType, ok := c.Types[fn]
if !ok || fnType.Type.Kind() != reflect.Func {
return fmt.Errorf("function %s for %s operator does not exist in environment", fn, op)
}
requiredNumIn := 2
if fnType.Method {
requiredNumIn = 3 // As first argument of method is receiver.
}
if fnType.Type.NumIn() != requiredNumIn || fnType.Type.NumOut() != 1 {
return fmt.Errorf("function %s for %s operator does not have a correct signature", fn, op)
}
}
}
// Check that all ConstExprFns are functions.
for name, fn := range c.ConstExprFns {
if fn.Kind() != reflect.Func {
return fmt.Errorf("const expression %q must be a function", name)
}
}
return c.err
}
func (c *Config) ConstExpr(name string) {
if c.Env == nil {
c.Error(fmt.Errorf("no environment for const expression: %v", name))
return
}
c.ConstExprFns[name] = vm.FetchFn(c.Env, name)
}
func (c *Config) Error(err error) {
if c.err == nil {
c.err = err
}
}
+26
View File
@@ -0,0 +1,26 @@
package conf
import "reflect"
// OperatorsTable maps binary operators to corresponding list of functions.
// Functions should be provided in the environment to allow operator overloading.
type OperatorsTable map[string][]string
func FindSuitableOperatorOverload(fns []string, types TypesTable, l, r reflect.Type) (reflect.Type, string, bool) {
for _, fn := range fns {
fnType := types[fn]
firstInIndex := 0
if fnType.Method {
firstInIndex = 1 // As first argument to method is receiver.
}
firstArgType := fnType.Type.In(firstInIndex)
secondArgType := fnType.Type.In(firstInIndex + 1)
firstArgumentFit := l == firstArgType || (firstArgType.Kind() == reflect.Interface && (l == nil || l.Implements(firstArgType)))
secondArgumentFit := r == secondArgType || (secondArgType.Kind() == reflect.Interface && (r == nil || r.Implements(secondArgType)))
if firstArgumentFit && secondArgumentFit {
return fnType.Type.Out(0), fn, true
}
}
return nil, "", false
}
+100
View File
@@ -0,0 +1,100 @@
package conf
import "reflect"
type Tag struct {
Type reflect.Type
Method bool
Ambiguous bool
}
type TypesTable map[string]Tag
// CreateTypesTable creates types table for type checks during parsing.
// If struct is passed, all fields will be treated as variables,
// as well as all fields of embedded structs and struct itself.
//
// If map is passed, all items will be treated as variables
// (key as name, value as type).
func CreateTypesTable(i interface{}) TypesTable {
if i == nil {
return nil
}
types := make(TypesTable)
v := reflect.ValueOf(i)
t := reflect.TypeOf(i)
d := t
if t.Kind() == reflect.Ptr {
d = t.Elem()
}
switch d.Kind() {
case reflect.Struct:
types = FieldsFromStruct(d)
// Methods of struct should be gathered from original struct with pointer,
// as methods maybe declared on pointer receiver. Also this method retrieves
// all embedded structs methods as well, no need to recursion.
for i := 0; i < t.NumMethod(); i++ {
m := t.Method(i)
types[m.Name] = Tag{Type: m.Type, Method: true}
}
case reflect.Map:
for _, key := range v.MapKeys() {
value := v.MapIndex(key)
if key.Kind() == reflect.String && value.IsValid() && value.CanInterface() {
types[key.String()] = Tag{Type: reflect.TypeOf(value.Interface())}
}
}
// A map may have method too.
for i := 0; i < t.NumMethod(); i++ {
m := t.Method(i)
types[m.Name] = Tag{Type: m.Type, Method: true}
}
}
return types
}
func FieldsFromStruct(t reflect.Type) TypesTable {
types := make(TypesTable)
t = dereference(t)
if t == nil {
return types
}
switch t.Kind() {
case reflect.Struct:
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
if f.Anonymous {
for name, typ := range FieldsFromStruct(f.Type) {
if _, ok := types[name]; ok {
types[name] = Tag{Ambiguous: true}
} else {
types[name] = typ
}
}
}
types[f.Name] = Tag{Type: f.Type}
}
}
return types
}
func dereference(t reflect.Type) reflect.Type {
if t == nil {
return nil
}
if t.Kind() == reflect.Ptr {
t = dereference(t.Elem())
}
return t
}
+187
View File
@@ -0,0 +1,187 @@
package expr
import (
"fmt"
"github.com/antonmedv/expr/ast"
"github.com/antonmedv/expr/file"
"reflect"
"github.com/antonmedv/expr/checker"
"github.com/antonmedv/expr/compiler"
"github.com/antonmedv/expr/conf"
"github.com/antonmedv/expr/optimizer"
"github.com/antonmedv/expr/parser"
"github.com/antonmedv/expr/vm"
)
// Option for configuring config.
type Option func(c *conf.Config)
// Eval parses, compiles and runs given input.
func Eval(input string, env interface{}) (interface{}, error) {
if _, ok := env.(Option); ok {
return nil, fmt.Errorf("misused expr.Eval: second argument (env) should be passed without expr.Env")
}
tree, err := parser.Parse(input)
if err != nil {
return nil, err
}
program, err := compiler.Compile(tree, nil)
if err != nil {
return nil, err
}
output, err := vm.Run(program, env)
if err != nil {
return nil, err
}
return output, nil
}
// Env specifies expected input of env for type checks.
// If struct is passed, all fields will be treated as variables,
// as well as all fields of embedded structs and struct itself.
// If map is passed, all items will be treated as variables.
// Methods defined on this type will be available as functions.
func Env(env interface{}) Option {
return func(c *conf.Config) {
if _, ok := env.(map[string]interface{}); ok {
c.MapEnv = true
} else {
if reflect.ValueOf(env).Kind() == reflect.Map {
c.DefaultType = reflect.TypeOf(env).Elem()
}
}
c.Strict = true
c.Types = conf.CreateTypesTable(env)
c.Env = env
}
}
// AllowUndefinedVariables allows to use undefined variables inside expressions.
// This can be used with expr.Env option to partially define a few variables.
// Note what this option is only works in map environment are used, otherwise
// runtime.fetch will panic as there is no way to get missing field zero value.
func AllowUndefinedVariables() Option {
return func(c *conf.Config) {
c.Strict = false
}
}
// Operator allows to override binary operator with function.
func Operator(operator string, fn ...string) Option {
return func(c *conf.Config) {
c.Operators[operator] = append(c.Operators[operator], fn...)
}
}
// ConstExpr defines func expression as constant. If all argument to this function is constants,
// then it can be replaced by result of this func call on compile step.
func ConstExpr(fn string) Option {
return func(c *conf.Config) {
c.ConstExpr(fn)
}
}
// AsBool tells the compiler to expect boolean result.
func AsBool() Option {
return func(c *conf.Config) {
c.Expect = reflect.Bool
}
}
// AsInt64 tells the compiler to expect int64 result.
func AsInt64() Option {
return func(c *conf.Config) {
c.Expect = reflect.Int64
}
}
// AsFloat64 tells the compiler to expect float64 result.
func AsFloat64() Option {
return func(c *conf.Config) {
c.Expect = reflect.Float64
}
}
// Optimize turns optimizations on or off.
func Optimize(b bool) Option {
return func(c *conf.Config) {
c.Optimize = b
}
}
// Patch adds visitor to list of visitors what will be applied before compiling AST to bytecode.
func Patch(visitor ast.Visitor) Option {
return func(c *conf.Config) {
c.Visitors = append(c.Visitors, visitor)
}
}
// Compile parses and compiles given input expression to bytecode program.
func Compile(input string, ops ...Option) (*vm.Program, error) {
config := &conf.Config{
Operators: make(map[string][]string),
ConstExprFns: make(map[string]reflect.Value),
Optimize: true,
}
for _, op := range ops {
op(config)
}
if err := config.Check(); err != nil {
return nil, err
}
tree, err := parser.Parse(input)
if err != nil {
return nil, err
}
_, err = checker.Check(tree, config)
// If we have a patch to apply, it may fix out error and
// second type check is needed. Otherwise it is an error.
if err != nil && len(config.Visitors) == 0 {
return nil, err
}
// Patch operators before Optimize, as we may also mark it as ConstExpr.
compiler.PatchOperators(&tree.Node, config)
if len(config.Visitors) >= 0 {
for _, v := range config.Visitors {
ast.Walk(&tree.Node, v)
}
_, err = checker.Check(tree, config)
if err != nil {
return nil, err
}
}
if config.Optimize {
err = optimizer.Optimize(&tree.Node, config)
if err != nil {
if fileError, ok := err.(*file.Error); ok {
return nil, fileError.Bind(tree.Source)
}
return nil, err
}
}
program, err := compiler.Compile(tree, config)
if err != nil {
return nil, err
}
return program, nil
}
// Run evaluates given bytecode program.
func Run(program *vm.Program, env interface{}) (interface{}, error) {
return vm.Run(program, env)
}
+58
View File
@@ -0,0 +1,58 @@
package file
import (
"fmt"
"strings"
"unicode/utf8"
)
type Error struct {
Location
Message string
Snippet string
}
func (e *Error) Error() string {
return e.format()
}
func (e *Error) Bind(source *Source) *Error {
if snippet, found := source.Snippet(e.Location.Line); found {
snippet := strings.Replace(snippet, "\t", " ", -1)
srcLine := "\n | " + snippet
var bytes = []byte(snippet)
var indLine = "\n | "
for i := 0; i < e.Location.Column && len(bytes) > 0; i++ {
_, sz := utf8.DecodeRune(bytes)
bytes = bytes[sz:]
if sz > 1 {
goto noind
} else {
indLine += "."
}
}
if _, sz := utf8.DecodeRune(bytes); sz > 1 {
goto noind
} else {
indLine += "^"
}
srcLine += indLine
noind:
e.Snippet = srcLine
}
return e
}
func (e *Error) format() string {
if e.Location.Empty() {
return e.Message
}
return fmt.Sprintf(
"%s (%d:%d)%s",
e.Message,
e.Line,
e.Column+1, // add one to the 0-based column for display
e.Snippet,
)
}
+10
View File
@@ -0,0 +1,10 @@
package file
type Location struct {
Line int // The 1-based line of the location.
Column int // The 0-based column number of the location.
}
func (l Location) Empty() bool {
return l.Column == 0 && l.Line == 0
}
+95
View File
@@ -0,0 +1,95 @@
package file
import (
"encoding/json"
"strings"
"unicode/utf8"
)
type Source struct {
contents []rune
lineOffsets []int32
}
func NewSource(contents string) *Source {
s := &Source{
contents: []rune(contents),
}
s.updateOffsets()
return s
}
func (s *Source) MarshalJSON() ([]byte, error) {
return json.Marshal(s.contents)
}
func (s *Source) UnmarshalJSON(b []byte) error {
contents := make([]rune, 0)
err := json.Unmarshal(b, &contents)
if err != nil {
return err
}
s.contents = contents
s.updateOffsets()
return nil
}
func (s *Source) Content() string {
return string(s.contents)
}
func (s *Source) Snippet(line int) (string, bool) {
charStart, found := s.findLineOffset(line)
if !found || len(s.contents) == 0 {
return "", false
}
charEnd, found := s.findLineOffset(line + 1)
if found {
return string(s.contents[charStart : charEnd-1]), true
}
return string(s.contents[charStart:]), true
}
// updateOffsets compute line offsets up front as they are referred to frequently.
func (s *Source) updateOffsets() {
lines := strings.Split(string(s.contents), "\n")
offsets := make([]int32, len(lines))
var offset int32
for i, line := range lines {
offset = offset + int32(utf8.RuneCountInString(line)) + 1
offsets[int32(i)] = offset
}
s.lineOffsets = offsets
}
// findLineOffset returns the offset where the (1-indexed) line begins,
// or false if line doesn't exist.
func (s *Source) findLineOffset(line int) (int32, bool) {
if line == 1 {
return 0, true
} else if line > 1 && line <= len(s.lineOffsets) {
offset := s.lineOffsets[line-2]
return offset, true
}
return -1, false
}
// findLine finds the line that contains the given character offset and
// returns the line number and offset of the beginning of that line.
// Note that the last line is treated as if it contains all offsets
// beyond the end of the actual source.
func (s *Source) findLine(characterOffset int32) (int32, int32) {
var line int32 = 1
for _, lineOffset := range s.lineOffsets {
if lineOffset > characterOffset {
break
} else {
line++
}
}
if line == 1 {
return line, 0
}
return line, s.lineOffsets[line-2]
}
+10
View File
@@ -0,0 +1,10 @@
module github.com/antonmedv/expr
go 1.13
require (
github.com/gdamore/tcell v1.3.0
github.com/rivo/tview v0.0.0-20200219210816-cd38d7432498
github.com/sanity-io/litter v1.2.0
github.com/stretchr/testify v1.5.1
)
+37
View File
@@ -0,0 +1,37 @@
github.com/DATA-DOG/go-sqlmock v1.3.3/go.mod h1:f/Ixk793poVmq4qj/V1dPUg2JEAKC73Q5eFN3EC/SaM=
github.com/davecgh/go-spew v0.0.0-20161028175848-04cdfd42973b/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/gdamore/encoding v1.0.0 h1:+7OoQ1Bc6eTm5niUzBa0Ctsh6JbMW6Ra+YNuAtDBdko=
github.com/gdamore/encoding v1.0.0/go.mod h1:alR0ol34c49FCSBLjhosxzcPHQbf2trDkoo5dl+VrEg=
github.com/gdamore/tcell v1.3.0 h1:r35w0JBADPZCVQijYebl6YMWWtHRqVEGt7kL2eBADRM=
github.com/gdamore/tcell v1.3.0/go.mod h1:Hjvr+Ofd+gLglo7RYKxxnzCBmev3BzsS67MebKS4zMM=
github.com/lucasb-eyer/go-colorful v1.0.2/go.mod h1:0MS4r+7BZKSJ5mw4/S5MPN+qHFF1fYclkSPilDOKW0s=
github.com/lucasb-eyer/go-colorful v1.0.3 h1:QIbQXiugsb+q10B+MI+7DI1oQLdmnep86tWFlaaUAac=
github.com/lucasb-eyer/go-colorful v1.0.3/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/mattn/go-runewidth v0.0.4/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.8 h1:3tS41NlGYSmhhe/8fhGRzc+z3AYCw1Fe1WAyLuujKs0=
github.com/mattn/go-runewidth v0.0.8/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/pmezard/go-difflib v0.0.0-20151028094244-d8ed2627bdf0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rivo/tview v0.0.0-20200219210816-cd38d7432498 h1:4CFNy7/q7P06AsIONZzuWy7jcdqEmYQvOZ9FAFZdbls=
github.com/rivo/tview v0.0.0-20200219210816-cd38d7432498/go.mod h1:6lkG1x+13OShEf0EaOCaTQYyB7d5nSbb181KtjlS+84=
github.com/rivo/uniseg v0.1.0 h1:+2KBaVoUmb9XzDsrx/Ct0W/EYOSFf/nWTauy++DprtY=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/sanity-io/litter v1.2.0 h1:DGJO0bxH/+C2EukzOSBmAlxmkhVMGqzvcx/rvySYw9M=
github.com/sanity-io/litter v1.2.0/go.mod h1:JF6pZUFgu2Q0sBZ+HSV35P8TVPI1TTzEwyu9FXAw2W4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v0.0.0-20161117074351-18a02ba4a312/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
golang.org/x/sys v0.0.0-20190626150813-e07cf5db2756/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4 h1:sfkvUWPNGwSV+8/fNqctR5lS2AqCSqYwXdrjCxp/dXo=
golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+77
View File
@@ -0,0 +1,77 @@
package optimizer
import (
"fmt"
. "github.com/antonmedv/expr/ast"
"github.com/antonmedv/expr/file"
"reflect"
"strings"
)
type constExpr struct {
applied bool
err error
fns map[string]reflect.Value
}
func (*constExpr) Enter(*Node) {}
func (c *constExpr) Exit(node *Node) {
defer func() {
if r := recover(); r != nil {
msg := fmt.Sprintf("%v", r)
// Make message more actual, it's a runtime error, but at compile step.
msg = strings.Replace(msg, "runtime error:", "compile error:", 1)
c.err = &file.Error{
Location: (*node).Location(),
Message: msg,
}
}
}()
patch := func(newNode Node) {
c.applied = true
Patch(node, newNode)
}
switch n := (*node).(type) {
case *FunctionNode:
fn, ok := c.fns[n.Name]
if ok {
in := make([]reflect.Value, len(n.Arguments))
for i := 0; i < len(n.Arguments); i++ {
arg := n.Arguments[i]
var param interface{}
switch a := arg.(type) {
case *NilNode:
param = nil
case *IntegerNode:
param = a.Value
case *FloatNode:
param = a.Value
case *BoolNode:
param = a.Value
case *StringNode:
param = a.Value
case *ConstantNode:
param = a.Value
default:
return // Const expr optimization not applicable.
}
if param == nil && reflect.TypeOf(param) == nil {
// In case of nil value and nil type use this hack,
// otherwise reflect.Call will panic on zero value.
in[i] = reflect.ValueOf(&param).Elem()
} else {
in[i] = reflect.ValueOf(param)
}
}
out := fn.Call(in)
constNode := &ConstantNode{Value: out[0].Interface()}
patch(constNode)
}
}
}
+33
View File
@@ -0,0 +1,33 @@
package optimizer
import (
. "github.com/antonmedv/expr/ast"
)
type constRange struct{}
func (*constRange) Enter(*Node) {}
func (*constRange) Exit(node *Node) {
switch n := (*node).(type) {
case *BinaryNode:
if n.Operator == ".." {
if min, ok := n.Left.(*IntegerNode); ok {
if max, ok := n.Right.(*IntegerNode); ok {
size := max.Value - min.Value + 1
// In this case array is too big. Skip generation,
// and wait for memory budget detection on runtime.
if size > 1e6 {
return
}
value := make([]int, size)
for i := range value {
value[i] = min.Value + i
}
Patch(node, &ConstantNode{
Value: value,
})
}
}
}
}
}
+133
View File
@@ -0,0 +1,133 @@
package optimizer
import (
"math"
"reflect"
. "github.com/antonmedv/expr/ast"
"github.com/antonmedv/expr/file"
)
type fold struct {
applied bool
err *file.Error
}
func (*fold) Enter(*Node) {}
func (fold *fold) Exit(node *Node) {
patch := func(newNode Node) {
fold.applied = true
Patch(node, newNode)
}
// for IntegerNode the type may have been changed from int->float
// preserve this information by setting the type after the Patch
patchWithType := func(newNode Node, leafType reflect.Type) {
patch(newNode)
newNode.SetType(leafType)
}
switch n := (*node).(type) {
case *UnaryNode:
switch n.Operator {
case "-":
if i, ok := n.Node.(*IntegerNode); ok {
patchWithType(&IntegerNode{Value: -i.Value}, n.Node.Type())
}
case "+":
if i, ok := n.Node.(*IntegerNode); ok {
patchWithType(&IntegerNode{Value: i.Value}, n.Node.Type())
}
}
case *BinaryNode:
switch n.Operator {
case "+":
if a, ok := n.Left.(*IntegerNode); ok {
if b, ok := n.Right.(*IntegerNode); ok {
patchWithType(&IntegerNode{Value: a.Value + b.Value}, a.Type())
}
}
if a, ok := n.Left.(*StringNode); ok {
if b, ok := n.Right.(*StringNode); ok {
patch(&StringNode{Value: a.Value + b.Value})
}
}
case "-":
if a, ok := n.Left.(*IntegerNode); ok {
if b, ok := n.Right.(*IntegerNode); ok {
patchWithType(&IntegerNode{Value: a.Value - b.Value}, a.Type())
}
}
case "*":
if a, ok := n.Left.(*IntegerNode); ok {
if b, ok := n.Right.(*IntegerNode); ok {
patchWithType(&IntegerNode{Value: a.Value * b.Value}, a.Type())
}
}
case "/":
if a, ok := n.Left.(*IntegerNode); ok {
if b, ok := n.Right.(*IntegerNode); ok {
if b.Value == 0 {
fold.err = &file.Error{
Location: (*node).Location(),
Message: "integer divide by zero",
}
return
}
patchWithType(&IntegerNode{Value: a.Value / b.Value}, a.Type())
}
}
case "%":
if a, ok := n.Left.(*IntegerNode); ok {
if b, ok := n.Right.(*IntegerNode); ok {
if b.Value == 0 {
fold.err = &file.Error{
Location: (*node).Location(),
Message: "integer divide by zero",
}
return
}
patch(&IntegerNode{Value: a.Value % b.Value})
}
}
case "**":
if a, ok := n.Left.(*IntegerNode); ok {
if b, ok := n.Right.(*IntegerNode); ok {
patch(&FloatNode{Value: math.Pow(float64(a.Value), float64(b.Value))})
}
}
}
case *ArrayNode:
if len(n.Nodes) > 0 {
for _, a := range n.Nodes {
if _, ok := a.(*IntegerNode); !ok {
goto string
}
}
{
value := make([]int, len(n.Nodes))
for i, a := range n.Nodes {
value[i] = a.(*IntegerNode).Value
}
patch(&ConstantNode{Value: value})
}
string:
for _, a := range n.Nodes {
if _, ok := a.(*StringNode); !ok {
return
}
}
{
value := make([]string, len(n.Nodes))
for i, a := range n.Nodes {
value[i] = a.(*StringNode).Value
}
patch(&ConstantNode{Value: value})
}
}
}
}
+65
View File
@@ -0,0 +1,65 @@
package optimizer
import (
"reflect"
. "github.com/antonmedv/expr/ast"
)
type inArray struct{}
func (*inArray) Enter(*Node) {}
func (*inArray) Exit(node *Node) {
switch n := (*node).(type) {
case *BinaryNode:
if n.Operator == "in" || n.Operator == "not in" {
if array, ok := n.Right.(*ArrayNode); ok {
if len(array.Nodes) > 0 {
t := n.Left.Type()
if t == nil || t.Kind() != reflect.Int {
// This optimization can be only performed if left side is int type,
// as runtime.in func uses reflect.Map.MapIndex and keys of map must,
// be same as checked value type.
goto string
}
for _, a := range array.Nodes {
if _, ok := a.(*IntegerNode); !ok {
goto string
}
}
{
value := make(map[int]struct{})
for _, a := range array.Nodes {
value[a.(*IntegerNode).Value] = struct{}{}
}
Patch(node, &BinaryNode{
Operator: n.Operator,
Left: n.Left,
Right: &ConstantNode{Value: value},
})
}
string:
for _, a := range array.Nodes {
if _, ok := a.(*StringNode); !ok {
return
}
}
{
value := make(map[string]struct{})
for _, a := range array.Nodes {
value[a.(*StringNode).Value] = struct{}{}
}
Patch(node, &BinaryNode{
Operator: n.Operator,
Left: n.Left,
Right: &ConstantNode{Value: value},
})
}
}
}
}
}
}
+41
View File
@@ -0,0 +1,41 @@
package optimizer
import (
. "github.com/antonmedv/expr/ast"
)
type inRange struct{}
func (*inRange) Enter(*Node) {}
func (*inRange) Exit(node *Node) {
switch n := (*node).(type) {
case *BinaryNode:
if n.Operator == "in" || n.Operator == "not in" {
if rng, ok := n.Right.(*BinaryNode); ok && rng.Operator == ".." {
if from, ok := rng.Left.(*IntegerNode); ok {
if to, ok := rng.Right.(*IntegerNode); ok {
Patch(node, &BinaryNode{
Operator: "and",
Left: &BinaryNode{
Operator: ">=",
Left: n.Left,
Right: from,
},
Right: &BinaryNode{
Operator: "<=",
Left: n.Left,
Right: to,
},
})
if n.Operator == "not in" {
Patch(node, &UnaryNode{
Operator: "not",
Node: *node,
})
}
}
}
}
}
}
}
+37
View File
@@ -0,0 +1,37 @@
package optimizer
import (
. "github.com/antonmedv/expr/ast"
"github.com/antonmedv/expr/conf"
)
func Optimize(node *Node, config *conf.Config) error {
Walk(node, &inArray{})
for limit := 1000; limit >= 0; limit-- {
fold := &fold{}
Walk(node, fold)
if fold.err != nil {
return fold.err
}
if !fold.applied {
break
}
}
if config != nil && len(config.ConstExprFns) > 0 {
for limit := 100; limit >= 0; limit-- {
constExpr := &constExpr{
fns: config.ConstExprFns,
}
Walk(node, constExpr)
if constExpr.err != nil {
return constExpr.err
}
if !constExpr.applied {
break
}
}
}
Walk(node, &inRange{})
Walk(node, &constRange{})
return nil
}
+205
View File
@@ -0,0 +1,205 @@
package lexer
import (
"fmt"
"strings"
"unicode/utf8"
"github.com/antonmedv/expr/file"
)
func Lex(source *file.Source) ([]Token, error) {
l := &lexer{
input: source.Content(),
tokens: make([]Token, 0),
}
l.loc = file.Location{1, 0}
l.prev = l.loc
l.startLoc = l.loc
for state := root; state != nil; {
state = state(l)
}
if l.err != nil {
return nil, l.err.Bind(source)
}
return l.tokens, nil
}
type lexer struct {
input string
state stateFn
tokens []Token
start, end int // current position in input
width int // last rune with
startLoc file.Location // start location
prev, loc file.Location // prev location of end location, end location
err *file.Error
}
const eof rune = -1
func (l *lexer) next() rune {
if l.end >= len(l.input) {
l.width = 0
return eof
}
r, w := utf8.DecodeRuneInString(l.input[l.end:])
l.width = w
l.end += w
l.prev = l.loc
if r == '\n' {
l.loc.Line++
l.loc.Column = 0
} else {
l.loc.Column++
}
return r
}
func (l *lexer) peek() rune {
r := l.next()
l.backup()
return r
}
func (l *lexer) backup() {
l.end -= l.width
l.loc = l.prev
}
func (l *lexer) emit(t Kind) {
l.emitValue(t, l.word())
}
func (l *lexer) emitValue(t Kind, value string) {
l.tokens = append(l.tokens, Token{
Location: l.startLoc,
Kind: t,
Value: value,
})
l.start = l.end
l.startLoc = l.loc
}
func (l *lexer) emitEOF() {
l.tokens = append(l.tokens, Token{
Location: l.prev, // Point to previous position for better error messages.
Kind: EOF,
})
l.start = l.end
l.startLoc = l.loc
}
func (l *lexer) word() string {
return l.input[l.start:l.end]
}
func (l *lexer) ignore() {
l.start = l.end
l.startLoc = l.loc
}
func (l *lexer) accept(valid string) bool {
if strings.ContainsRune(valid, l.next()) {
return true
}
l.backup()
return false
}
func (l *lexer) acceptRun(valid string) {
for strings.ContainsRune(valid, l.next()) {
}
l.backup()
}
func (l *lexer) acceptWord(word string) bool {
pos := l.end
loc := l.loc
prev := l.prev
for _, ch := range word {
if l.next() != ch {
l.end = pos
l.loc = loc
l.prev = prev
return false
}
}
return true
}
func (l *lexer) error(format string, args ...interface{}) stateFn {
if l.err == nil { // show first error
l.err = &file.Error{
Location: l.loc,
Message: fmt.Sprintf(format, args...),
}
}
return nil
}
func digitVal(ch rune) int {
switch {
case '0' <= ch && ch <= '9':
return int(ch - '0')
case 'a' <= lower(ch) && lower(ch) <= 'f':
return int(lower(ch) - 'a' + 10)
}
return 16 // larger than any legal digit val
}
func lower(ch rune) rune { return ('a' - 'A') | ch } // returns lower-case ch iff ch is ASCII letter
func (l *lexer) scanDigits(ch rune, base, n int) rune {
for n > 0 && digitVal(ch) < base {
ch = l.next()
n--
}
if n > 0 {
l.error("invalid char escape")
}
return ch
}
func (l *lexer) scanEscape(quote rune) rune {
ch := l.next() // read character after '/'
switch ch {
case 'a', 'b', 'f', 'n', 'r', 't', 'v', '\\', quote:
// nothing to do
ch = l.next()
case '0', '1', '2', '3', '4', '5', '6', '7':
ch = l.scanDigits(ch, 8, 3)
case 'x':
ch = l.scanDigits(l.next(), 16, 2)
case 'u':
ch = l.scanDigits(l.next(), 16, 4)
case 'U':
ch = l.scanDigits(l.next(), 16, 8)
default:
l.error("invalid char escape")
}
return ch
}
func (l *lexer) scanString(quote rune) (n int) {
ch := l.next() // read character after quote
for ch != quote {
if ch == '\n' || ch == eof {
l.error("literal not terminated")
return
}
if ch == '\\' {
ch = l.scanEscape(quote)
} else {
ch = l.next()
}
n++
}
return
}
+134
View File
@@ -0,0 +1,134 @@
package lexer
import (
"strings"
)
type stateFn func(*lexer) stateFn
func root(l *lexer) stateFn {
switch r := l.next(); {
case r == eof:
l.emitEOF()
return nil
case IsSpace(r):
l.ignore()
return root
case r == '\'' || r == '"':
l.scanString(r)
str, err := unescape(l.word())
if err != nil {
l.error("%v", err)
}
l.emitValue(String, str)
case '0' <= r && r <= '9':
l.backup()
return number
case strings.ContainsRune("([{", r):
l.emit(Bracket)
case strings.ContainsRune(")]}", r):
l.emit(Bracket)
case strings.ContainsRune("#,?:%+-/", r): // single rune operator
l.emit(Operator)
case strings.ContainsRune("&|!=*<>", r): // possible double rune operator
l.accept("&|=*")
l.emit(Operator)
case r == '.':
l.backup()
return dot
case IsAlphaNumeric(r):
l.backup()
return identifier
default:
return l.error("unrecognized character: %#U", r)
}
return root
}
func number(l *lexer) stateFn {
if !l.scanNumber() {
return l.error("bad number syntax: %q", l.word())
}
l.emit(Number)
return root
}
func (l *lexer) scanNumber() bool {
digits := "0123456789_"
// Is it hex?
if l.accept("0") {
// Note: Leading 0 does not mean octal in floats.
if l.accept("xX") {
digits = "0123456789abcdefABCDEF_"
} else if l.accept("oO") {
digits = "01234567_"
} else if l.accept("bB") {
digits = "01_"
}
}
l.acceptRun(digits)
loc, prev, end := l.loc, l.prev, l.end
if l.accept(".") {
// Lookup for .. operator: if after dot there is another dot (1..2), it maybe a range operator.
if l.peek() == '.' {
// We can't backup() here, as it would require two backups,
// and backup() func supports only one for now. So, save and
// restore it here.
l.loc, l.prev, l.end = loc, prev, end
return true
}
l.acceptRun(digits)
}
if l.accept("eE") {
l.accept("+-")
l.acceptRun(digits)
}
// Next thing mustn't be alphanumeric.
if IsAlphaNumeric(l.peek()) {
l.next()
return false
}
return true
}
func dot(l *lexer) stateFn {
l.next()
if l.accept("0123456789") {
l.backup()
return number
}
l.accept(".")
l.emit(Operator)
return root
}
func identifier(l *lexer) stateFn {
loop:
for {
switch r := l.next(); {
case IsAlphaNumeric(r):
// absorb
default:
l.backup()
switch l.word() {
case "not":
return not
case "in", "or", "and", "matches", "contains", "startsWith", "endsWith":
l.emit(Operator)
default:
l.emit(Identifier)
}
break loop
}
}
return root
}
func not(l *lexer) stateFn {
if l.acceptWord(" in") {
l.emit(Operator)
} else {
l.emit(Operator)
}
return root
}
+47
View File
@@ -0,0 +1,47 @@
package lexer
import (
"fmt"
"github.com/antonmedv/expr/file"
)
type Kind string
const (
Identifier Kind = "Identifier"
Number = "Number"
String = "String"
Operator = "Operator"
Bracket = "Bracket"
EOF = "EOF"
)
type Token struct {
file.Location
Kind Kind
Value string
}
func (t Token) String() string {
if t.Value == "" {
return string(t.Kind)
}
return fmt.Sprintf("%s(%#v)", t.Kind, t.Value)
}
func (t Token) Is(kind Kind, values ...string) bool {
if len(values) == 0 {
return kind == t.Kind
}
for _, v := range values {
if v == t.Value {
goto found
}
}
return false
found:
return kind == t.Kind
}
+194
View File
@@ -0,0 +1,194 @@
package lexer
import (
"fmt"
"strings"
"unicode"
"unicode/utf8"
)
func IsSpace(r rune) bool {
return unicode.IsSpace(r)
}
func IsAlphaNumeric(r rune) bool {
return IsAlphabetic(r) || unicode.IsDigit(r)
}
func IsAlphabetic(r rune) bool {
return r == '_' || r == '$' || unicode.IsLetter(r)
}
var (
newlineNormalizer = strings.NewReplacer("\r\n", "\n", "\r", "\n")
)
// Unescape takes a quoted string, unquotes, and unescapes it.
func unescape(value string) (string, error) {
// All strings normalize newlines to the \n representation.
value = newlineNormalizer.Replace(value)
n := len(value)
// Nothing to unescape / decode.
if n < 2 {
return value, fmt.Errorf("unable to unescape string")
}
// Quoted string of some form, must have same first and last char.
if value[0] != value[n-1] || (value[0] != '"' && value[0] != '\'') {
return value, fmt.Errorf("unable to unescape string")
}
value = value[1 : n-1]
// The string contains escape characters.
// The following logic is adapted from `strconv/quote.go`
var runeTmp [utf8.UTFMax]byte
buf := make([]byte, 0, 3*n/2)
for len(value) > 0 {
c, multibyte, rest, err := unescapeChar(value)
if err != nil {
return "", err
}
value = rest
if c < utf8.RuneSelf || !multibyte {
buf = append(buf, byte(c))
} else {
n := utf8.EncodeRune(runeTmp[:], c)
buf = append(buf, runeTmp[:n]...)
}
}
return string(buf), nil
}
// unescapeChar takes a string input and returns the following info:
//
// value - the escaped unicode rune at the front of the string.
// multibyte - whether the rune value might require multiple bytes to represent.
// tail - the remainder of the input string.
// err - error value, if the character could not be unescaped.
//
// When multibyte is true the return value may still fit within a single byte,
// but a multibyte conversion is attempted which is more expensive than when the
// value is known to fit within one byte.
func unescapeChar(s string) (value rune, multibyte bool, tail string, err error) {
// 1. Character is not an escape sequence.
switch c := s[0]; {
case c >= utf8.RuneSelf:
r, size := utf8.DecodeRuneInString(s)
return r, true, s[size:], nil
case c != '\\':
return rune(s[0]), false, s[1:], nil
}
// 2. Last character is the start of an escape sequence.
if len(s) <= 1 {
err = fmt.Errorf("unable to unescape string, found '\\' as last character")
return
}
c := s[1]
s = s[2:]
// 3. Common escape sequences shared with Google SQL
switch c {
case 'a':
value = '\a'
case 'b':
value = '\b'
case 'f':
value = '\f'
case 'n':
value = '\n'
case 'r':
value = '\r'
case 't':
value = '\t'
case 'v':
value = '\v'
case '\\':
value = '\\'
case '\'':
value = '\''
case '"':
value = '"'
case '`':
value = '`'
case '?':
value = '?'
// 4. Unicode escape sequences, reproduced from `strconv/quote.go`
case 'x', 'X', 'u', 'U':
n := 0
switch c {
case 'x', 'X':
n = 2
case 'u':
n = 4
case 'U':
n = 8
}
var v rune
if len(s) < n {
err = fmt.Errorf("unable to unescape string")
return
}
for j := 0; j < n; j++ {
x, ok := unhex(s[j])
if !ok {
err = fmt.Errorf("unable to unescape string")
return
}
v = v<<4 | x
}
s = s[n:]
if v > utf8.MaxRune {
err = fmt.Errorf("unable to unescape string")
return
}
value = v
multibyte = true
// 5. Octal escape sequences, must be three digits \[0-3][0-7][0-7]
case '0', '1', '2', '3':
if len(s) < 2 {
err = fmt.Errorf("unable to unescape octal sequence in string")
return
}
v := rune(c - '0')
for j := 0; j < 2; j++ {
x := s[j]
if x < '0' || x > '7' {
err = fmt.Errorf("unable to unescape octal sequence in string")
return
}
v = v*8 + rune(x-'0')
}
if v > utf8.MaxRune {
err = fmt.Errorf("unable to unescape string")
return
}
value = v
s = s[2:]
multibyte = true
// Unknown escape sequence.
default:
err = fmt.Errorf("unable to unescape string")
}
tail = s
return
}
func unhex(b byte) (rune, bool) {
c := rune(b)
switch {
case '0' <= c && c <= '9':
return c - '0', true
case 'a' <= c && c <= 'f':
return c - 'a' + 10, true
case 'A' <= c && c <= 'F':
return c - 'A' + 10, true
}
return 0, false
}
+579
View File
@@ -0,0 +1,579 @@
package parser
import (
"fmt"
"regexp"
"strconv"
"strings"
"unicode/utf8"
. "github.com/antonmedv/expr/ast"
"github.com/antonmedv/expr/file"
. "github.com/antonmedv/expr/parser/lexer"
)
type associativity int
const (
left associativity = iota + 1
right
)
type operator struct {
precedence int
associativity associativity
}
type builtin struct {
arity int
}
var unaryOperators = map[string]operator{
"not": {50, left},
"!": {50, left},
"-": {500, left},
"+": {500, left},
}
var binaryOperators = map[string]operator{
"or": {10, left},
"||": {10, left},
"and": {15, left},
"&&": {15, left},
"==": {20, left},
"!=": {20, left},
"<": {20, left},
">": {20, left},
">=": {20, left},
"<=": {20, left},
"not in": {20, left},
"in": {20, left},
"matches": {20, left},
"contains": {20, left},
"startsWith": {20, left},
"endsWith": {20, left},
"..": {25, left},
"+": {30, left},
"-": {30, left},
"*": {60, left},
"/": {60, left},
"%": {60, left},
"**": {70, right},
}
var builtins = map[string]builtin{
"len": {1},
"all": {2},
"none": {2},
"any": {2},
"one": {2},
"filter": {2},
"map": {2},
"count": {2},
}
type parser struct {
tokens []Token
current Token
pos int
err *file.Error
depth int // closure call depth
}
type Tree struct {
Node Node
Source *file.Source
}
func Parse(input string) (*Tree, error) {
source := file.NewSource(input)
tokens, err := Lex(source)
if err != nil {
return nil, err
}
p := &parser{
tokens: tokens,
current: tokens[0],
}
node := p.parseExpression(0)
if !p.current.Is(EOF) {
p.error("unexpected token %v", p.current)
}
if p.err != nil {
return nil, p.err.Bind(source)
}
return &Tree{
Node: node,
Source: source,
}, nil
}
func (p *parser) error(format string, args ...interface{}) {
if p.err == nil { // show first error
p.err = &file.Error{
Location: p.current.Location,
Message: fmt.Sprintf(format, args...),
}
}
}
func (p *parser) next() {
p.pos++
if p.pos >= len(p.tokens) {
p.error("unexpected end of expression")
return
}
p.current = p.tokens[p.pos]
}
func (p *parser) expect(kind Kind, values ...string) {
if p.current.Is(kind, values...) {
p.next()
return
}
p.error("unexpected token %v", p.current)
}
// parse functions
func (p *parser) parseExpression(precedence int) Node {
nodeLeft := p.parsePrimary()
token := p.current
for token.Is(Operator) && p.err == nil {
if op, ok := binaryOperators[token.Value]; ok {
if op.precedence >= precedence {
p.next()
var nodeRight Node
if op.associativity == left {
nodeRight = p.parseExpression(op.precedence + 1)
} else {
nodeRight = p.parseExpression(op.precedence)
}
if token.Is(Operator, "matches") {
var r *regexp.Regexp
var err error
if s, ok := nodeRight.(*StringNode); ok {
r, err = regexp.Compile(s.Value)
if err != nil {
p.error("%v", err)
}
}
nodeLeft = &MatchesNode{
Regexp: r,
Left: nodeLeft,
Right: nodeRight,
}
nodeLeft.SetLocation(token.Location)
} else {
nodeLeft = &BinaryNode{
Operator: token.Value,
Left: nodeLeft,
Right: nodeRight,
}
nodeLeft.SetLocation(token.Location)
}
token = p.current
continue
}
}
break
}
if precedence == 0 {
nodeLeft = p.parseConditionalExpression(nodeLeft)
}
return nodeLeft
}
func (p *parser) parsePrimary() Node {
token := p.current
if token.Is(Operator) {
if op, ok := unaryOperators[token.Value]; ok {
p.next()
expr := p.parseExpression(op.precedence)
node := &UnaryNode{
Operator: token.Value,
Node: expr,
}
node.SetLocation(token.Location)
return p.parsePostfixExpression(node)
}
}
if token.Is(Bracket, "(") {
p.next()
expr := p.parseExpression(0)
p.expect(Bracket, ")") // "an opened parenthesis is not properly closed"
return p.parsePostfixExpression(expr)
}
if p.depth > 0 {
if token.Is(Operator, "#") || token.Is(Operator, ".") {
if token.Is(Operator, "#") {
p.next()
}
node := &PointerNode{}
node.SetLocation(token.Location)
return p.parsePostfixExpression(node)
}
} else {
if token.Is(Operator, "#") || token.Is(Operator, ".") {
p.error("cannot use pointer accessor outside closure")
}
}
return p.parsePrimaryExpression()
}
func (p *parser) parseConditionalExpression(node Node) Node {
var expr1, expr2 Node
for p.current.Is(Operator, "?") && p.err == nil {
p.next()
if !p.current.Is(Operator, ":") {
expr1 = p.parseExpression(0)
p.expect(Operator, ":")
expr2 = p.parseExpression(0)
} else {
p.next()
expr1 = node
expr2 = p.parseExpression(0)
}
node = &ConditionalNode{
Cond: node,
Exp1: expr1,
Exp2: expr2,
}
}
return node
}
func (p *parser) parsePrimaryExpression() Node {
var node Node
token := p.current
switch token.Kind {
case Identifier:
p.next()
switch token.Value {
case "true":
node := &BoolNode{Value: true}
node.SetLocation(token.Location)
return node
case "false":
node := &BoolNode{Value: false}
node.SetLocation(token.Location)
return node
case "nil":
node := &NilNode{}
node.SetLocation(token.Location)
return node
default:
node = p.parseIdentifierExpression(token)
}
case Number:
p.next()
value := strings.Replace(token.Value, "_", "", -1)
if strings.ContainsAny(value, ".eE") {
number, err := strconv.ParseFloat(value, 64)
if err != nil {
p.error("invalid float literal: %v", err)
}
node := &FloatNode{Value: number}
node.SetLocation(token.Location)
return node
} else if strings.Contains(value, "x") {
number, err := strconv.ParseInt(value, 0, 64)
if err != nil {
p.error("invalid hex literal: %v", err)
}
node := &IntegerNode{Value: int(number)}
node.SetLocation(token.Location)
return node
} else {
number, err := strconv.ParseInt(value, 10, 64)
if err != nil {
p.error("invalid integer literal: %v", err)
}
node := &IntegerNode{Value: int(number)}
node.SetLocation(token.Location)
return node
}
case String:
p.next()
node := &StringNode{Value: token.Value}
node.SetLocation(token.Location)
return node
default:
if token.Is(Bracket, "[") {
node = p.parseArrayExpression(token)
} else if token.Is(Bracket, "{") {
node = p.parseMapExpression(token)
} else {
p.error("unexpected token %v", token)
}
}
return p.parsePostfixExpression(node)
}
func (p *parser) parseIdentifierExpression(token Token) Node {
var node Node
if p.current.Is(Bracket, "(") {
var arguments []Node
if b, ok := builtins[token.Value]; ok {
p.expect(Bracket, "(")
// TODO: Add builtins signatures.
if b.arity == 1 {
arguments = make([]Node, 1)
arguments[0] = p.parseExpression(0)
} else if b.arity == 2 {
arguments = make([]Node, 2)
arguments[0] = p.parseExpression(0)
p.expect(Operator, ",")
arguments[1] = p.parseClosure()
}
p.expect(Bracket, ")")
node = &BuiltinNode{
Name: token.Value,
Arguments: arguments,
}
node.SetLocation(token.Location)
} else {
arguments = p.parseArguments()
node = &FunctionNode{
Name: token.Value,
Arguments: arguments,
}
node.SetLocation(token.Location)
}
} else {
node = &IdentifierNode{Value: token.Value}
node.SetLocation(token.Location)
}
return node
}
func (p *parser) parseClosure() Node {
token := p.current
p.expect(Bracket, "{")
p.depth++
node := p.parseExpression(0)
p.depth--
p.expect(Bracket, "}")
closure := &ClosureNode{
Node: node,
}
closure.SetLocation(token.Location)
return closure
}
func (p *parser) parseArrayExpression(token Token) Node {
nodes := make([]Node, 0)
p.expect(Bracket, "[")
for !p.current.Is(Bracket, "]") && p.err == nil {
if len(nodes) > 0 {
p.expect(Operator, ",")
if p.current.Is(Bracket, "]") {
goto end
}
}
node := p.parseExpression(0)
nodes = append(nodes, node)
}
end:
p.expect(Bracket, "]")
node := &ArrayNode{Nodes: nodes}
node.SetLocation(token.Location)
return node
}
func (p *parser) parseMapExpression(token Token) Node {
p.expect(Bracket, "{")
nodes := make([]Node, 0)
for !p.current.Is(Bracket, "}") && p.err == nil {
if len(nodes) > 0 {
p.expect(Operator, ",")
if p.current.Is(Bracket, "}") {
goto end
}
if p.current.Is(Operator, ",") {
p.error("unexpected token %v", p.current)
}
}
var key Node
// a map key can be:
// * a number
// * a string
// * a identifier, which is equivalent to a string
// * an expression, which must be enclosed in parentheses -- (1 + 2)
if p.current.Is(Number) || p.current.Is(String) || p.current.Is(Identifier) {
key = &StringNode{Value: p.current.Value}
key.SetLocation(token.Location)
p.next()
} else if p.current.Is(Bracket, "(") {
key = p.parseExpression(0)
} else {
p.error("a map key must be a quoted string, a number, a identifier, or an expression enclosed in parentheses (unexpected token %v)", p.current)
}
p.expect(Operator, ":")
node := p.parseExpression(0)
pair := &PairNode{Key: key, Value: node}
pair.SetLocation(token.Location)
nodes = append(nodes, pair)
}
end:
p.expect(Bracket, "}")
node := &MapNode{Pairs: nodes}
node.SetLocation(token.Location)
return node
}
func (p *parser) parsePostfixExpression(node Node) Node {
token := p.current
for (token.Is(Operator) || token.Is(Bracket)) && p.err == nil {
if token.Value == "." {
p.next()
token = p.current
p.next()
if token.Kind != Identifier &&
// Operators like "not" and "matches" are valid methods or property names.
(token.Kind != Operator || !isValidIdentifier(token.Value)) {
p.error("expected name")
}
if p.current.Is(Bracket, "(") {
arguments := p.parseArguments()
node = &MethodNode{
Node: node,
Method: token.Value,
Arguments: arguments,
}
node.SetLocation(token.Location)
} else {
node = &PropertyNode{
Node: node,
Property: token.Value,
}
node.SetLocation(token.Location)
}
} else if token.Value == "[" {
p.next()
var from, to Node
if p.current.Is(Operator, ":") { // slice without from [:1]
p.next()
if !p.current.Is(Bracket, "]") { // slice without from and to [:]
to = p.parseExpression(0)
}
node = &SliceNode{
Node: node,
To: to,
}
node.SetLocation(token.Location)
p.expect(Bracket, "]")
} else {
from = p.parseExpression(0)
if p.current.Is(Operator, ":") {
p.next()
if !p.current.Is(Bracket, "]") { // slice without to [1:]
to = p.parseExpression(0)
}
node = &SliceNode{
Node: node,
From: from,
To: to,
}
node.SetLocation(token.Location)
p.expect(Bracket, "]")
} else {
// Slice operator [:] was not found, it should by just index node.
node = &IndexNode{
Node: node,
Index: from,
}
node.SetLocation(token.Location)
p.expect(Bracket, "]")
}
}
} else {
break
}
token = p.current
}
return node
}
func isValidIdentifier(str string) bool {
if len(str) == 0 {
return false
}
h, w := utf8.DecodeRuneInString(str)
if !IsAlphabetic(h) {
return false
}
for _, r := range str[w:] {
if !IsAlphaNumeric(r) {
return false
}
}
return true
}
func (p *parser) parseArguments() []Node {
p.expect(Bracket, "(")
nodes := make([]Node, 0)
for !p.current.Is(Bracket, ")") && p.err == nil {
if len(nodes) > 0 {
p.expect(Operator, ",")
}
node := p.parseExpression(0)
nodes = append(nodes, node)
}
p.expect(Bracket, ")")
return nodes
}
+3247
View File
File diff suppressed because it is too large Load Diff
+53
View File
@@ -0,0 +1,53 @@
package vm
const (
OpPush byte = iota
OpPop
OpRot
OpFetch
OpFetchMap
OpTrue
OpFalse
OpNil
OpNegate
OpNot
OpEqual
OpEqualInt
OpEqualString
OpJump
OpJumpIfTrue
OpJumpIfFalse
OpJumpBackward
OpIn
OpLess
OpMore
OpLessOrEqual
OpMoreOrEqual
OpAdd
OpSubtract
OpMultiply
OpDivide
OpModulo
OpExponent
OpRange
OpMatches
OpMatchesConst
OpContains
OpStartsWith
OpEndsWith
OpIndex
OpSlice
OpProperty
OpCall
OpCallFast
OpMethod
OpArray
OpMap
OpLen
OpCast
OpStore
OpLoad
OpInc
OpBegin
OpEnd // This opcode must be at the end of this list.
)
+216
View File
@@ -0,0 +1,216 @@
package vm
import (
"encoding/binary"
"fmt"
"regexp"
"github.com/antonmedv/expr/file"
)
type Program struct {
Source *file.Source
Locations map[int]file.Location
Constants []interface{}
Bytecode []byte
}
func (program *Program) Disassemble() string {
out := ""
ip := 0
for ip < len(program.Bytecode) {
pp := ip
op := program.Bytecode[ip]
ip++
readArg := func() uint16 {
if ip+1 >= len(program.Bytecode) {
return 0
}
i := binary.LittleEndian.Uint16([]byte{program.Bytecode[ip], program.Bytecode[ip+1]})
ip += 2
return i
}
code := func(label string) {
out += fmt.Sprintf("%v\t%v\n", pp, label)
}
jump := func(label string) {
a := readArg()
out += fmt.Sprintf("%v\t%v\t%v\t(%v)\n", pp, label, a, ip+int(a))
}
back := func(label string) {
a := readArg()
out += fmt.Sprintf("%v\t%v\t%v\t(%v)\n", pp, label, a, ip-int(a))
}
argument := func(label string) {
a := readArg()
out += fmt.Sprintf("%v\t%v\t%v\n", pp, label, a)
}
constant := func(label string) {
a := readArg()
var c interface{}
if int(a) < len(program.Constants) {
c = program.Constants[a]
}
if r, ok := c.(*regexp.Regexp); ok {
c = r.String()
}
out += fmt.Sprintf("%v\t%v\t%v\t%#v\n", pp, label, a, c)
}
switch op {
case OpPush:
constant("OpPush")
case OpPop:
code("OpPop")
case OpRot:
code("OpRot")
case OpFetch:
constant("OpFetch")
case OpFetchMap:
constant("OpFetchMap")
case OpTrue:
code("OpTrue")
case OpFalse:
code("OpFalse")
case OpNil:
code("OpNil")
case OpNegate:
code("OpNegate")
case OpNot:
code("OpNot")
case OpEqual:
code("OpEqual")
case OpEqualInt:
code("OpEqualInt")
case OpEqualString:
code("OpEqualString")
case OpJump:
jump("OpJump")
case OpJumpIfTrue:
jump("OpJumpIfTrue")
case OpJumpIfFalse:
jump("OpJumpIfFalse")
case OpJumpBackward:
back("OpJumpBackward")
case OpIn:
code("OpIn")
case OpLess:
code("OpLess")
case OpMore:
code("OpMore")
case OpLessOrEqual:
code("OpLessOrEqual")
case OpMoreOrEqual:
code("OpMoreOrEqual")
case OpAdd:
code("OpAdd")
case OpSubtract:
code("OpSubtract")
case OpMultiply:
code("OpMultiply")
case OpDivide:
code("OpDivide")
case OpModulo:
code("OpModulo")
case OpExponent:
code("OpExponent")
case OpRange:
code("OpRange")
case OpMatches:
code("OpMatches")
case OpMatchesConst:
constant("OpMatchesConst")
case OpContains:
code("OpContains")
case OpStartsWith:
code("OpStartsWith")
case OpEndsWith:
code("OpEndsWith")
case OpIndex:
code("OpIndex")
case OpSlice:
code("OpSlice")
case OpProperty:
constant("OpProperty")
case OpCall:
constant("OpCall")
case OpCallFast:
constant("OpCallFast")
case OpMethod:
constant("OpMethod")
case OpArray:
code("OpArray")
case OpMap:
code("OpMap")
case OpLen:
code("OpLen")
case OpCast:
argument("OpCast")
case OpStore:
constant("OpStore")
case OpLoad:
constant("OpLoad")
case OpInc:
constant("OpInc")
case OpBegin:
code("OpBegin")
case OpEnd:
code("OpEnd")
default:
out += fmt.Sprintf("%v\t%#x\n", pp, op)
}
}
return out
}
+343
View File
@@ -0,0 +1,343 @@
package vm
//go:generate go run ./generate
import (
"fmt"
"math"
"reflect"
)
type Call struct {
Name string
Size int
}
type Scope map[string]interface{}
func fetch(from interface{}, i interface{}) interface{} {
v := reflect.ValueOf(from)
kind := v.Kind()
// Structures can be access through a pointer or through a value, when they
// are accessed through a pointer we don't want to copy them to a value.
if kind == reflect.Ptr && reflect.Indirect(v).Kind() == reflect.Struct {
v = reflect.Indirect(v)
kind = v.Kind()
}
switch kind {
case reflect.Array, reflect.Slice, reflect.String:
value := v.Index(toInt(i))
if value.IsValid() && value.CanInterface() {
return value.Interface()
}
case reflect.Map:
value := v.MapIndex(reflect.ValueOf(i))
if value.IsValid() {
if value.CanInterface() {
return value.Interface()
}
} else {
elem := reflect.TypeOf(from).Elem()
return reflect.Zero(elem).Interface()
}
case reflect.Struct:
value := v.FieldByName(reflect.ValueOf(i).String())
if value.IsValid() && value.CanInterface() {
return value.Interface()
}
}
panic(fmt.Sprintf("cannot fetch %v from %T", i, from))
}
func slice(array, from, to interface{}) interface{} {
v := reflect.ValueOf(array)
switch v.Kind() {
case reflect.Array, reflect.Slice, reflect.String:
length := v.Len()
a, b := toInt(from), toInt(to)
if b > length {
b = length
}
if a > b {
a = b
}
value := v.Slice(a, b)
if value.IsValid() && value.CanInterface() {
return value.Interface()
}
case reflect.Ptr:
value := v.Elem()
if value.IsValid() && value.CanInterface() {
return slice(value.Interface(), from, to)
}
}
panic(fmt.Sprintf("cannot slice %v", from))
}
func FetchFn(from interface{}, name string) reflect.Value {
v := reflect.ValueOf(from)
// Methods can be defined on any type.
if v.NumMethod() > 0 {
method := v.MethodByName(name)
if method.IsValid() {
return method
}
}
d := v
if v.Kind() == reflect.Ptr {
d = v.Elem()
}
switch d.Kind() {
case reflect.Map:
value := d.MapIndex(reflect.ValueOf(name))
if value.IsValid() && value.CanInterface() {
return value.Elem()
}
case reflect.Struct:
// If struct has not method, maybe it has func field.
// To access this field we need dereference value.
value := d.FieldByName(name)
if value.IsValid() {
return value
}
}
panic(fmt.Sprintf(`cannot get "%v" from %T`, name, from))
}
func in(needle interface{}, array interface{}) bool {
if array == nil {
return false
}
v := reflect.ValueOf(array)
switch v.Kind() {
case reflect.Array, reflect.Slice:
for i := 0; i < v.Len(); i++ {
value := v.Index(i)
if value.IsValid() && value.CanInterface() {
if equal(value.Interface(), needle).(bool) {
return true
}
}
}
return false
case reflect.Map:
n := reflect.ValueOf(needle)
if !n.IsValid() {
panic(fmt.Sprintf("cannot use %T as index to %T", needle, array))
}
value := v.MapIndex(n)
if value.IsValid() {
return true
}
return false
case reflect.Struct:
n := reflect.ValueOf(needle)
if !n.IsValid() || n.Kind() != reflect.String {
panic(fmt.Sprintf("cannot use %T as field name of %T", needle, array))
}
value := v.FieldByName(n.String())
if value.IsValid() {
return true
}
return false
case reflect.Ptr:
value := v.Elem()
if value.IsValid() && value.CanInterface() {
return in(needle, value.Interface())
}
return false
}
panic(fmt.Sprintf(`operator "in"" not defined on %T`, array))
}
func length(a interface{}) int {
v := reflect.ValueOf(a)
switch v.Kind() {
case reflect.Array, reflect.Slice, reflect.Map, reflect.String:
return v.Len()
default:
panic(fmt.Sprintf("invalid argument for len (type %T)", a))
}
}
func negate(i interface{}) interface{} {
switch v := i.(type) {
case float32:
return -v
case float64:
return -v
case int:
return -v
case int8:
return -v
case int16:
return -v
case int32:
return -v
case int64:
return -v
case uint:
return -v
case uint8:
return -v
case uint16:
return -v
case uint32:
return -v
case uint64:
return -v
default:
panic(fmt.Sprintf("invalid operation: - %T", v))
}
}
func exponent(a, b interface{}) float64 {
return math.Pow(toFloat64(a), toFloat64(b))
}
func makeRange(min, max int) []int {
size := max - min + 1
rng := make([]int, size)
for i := range rng {
rng[i] = min + i
}
return rng
}
func toInt(a interface{}) int {
switch x := a.(type) {
case float32:
return int(x)
case float64:
return int(x)
case int:
return x
case int8:
return int(x)
case int16:
return int(x)
case int32:
return int(x)
case int64:
return int(x)
case uint:
return int(x)
case uint8:
return int(x)
case uint16:
return int(x)
case uint32:
return int(x)
case uint64:
return int(x)
default:
panic(fmt.Sprintf("invalid operation: int(%T)", x))
}
}
func toInt64(a interface{}) int64 {
switch x := a.(type) {
case float32:
return int64(x)
case float64:
return int64(x)
case int:
return int64(x)
case int8:
return int64(x)
case int16:
return int64(x)
case int32:
return int64(x)
case int64:
return x
case uint:
return int64(x)
case uint8:
return int64(x)
case uint16:
return int64(x)
case uint32:
return int64(x)
case uint64:
return int64(x)
default:
panic(fmt.Sprintf("invalid operation: int64(%T)", x))
}
}
func toFloat64(a interface{}) float64 {
switch x := a.(type) {
case float32:
return float64(x)
case float64:
return x
case int:
return float64(x)
case int8:
return float64(x)
case int16:
return float64(x)
case int32:
return float64(x)
case int64:
return float64(x)
case uint:
return float64(x)
case uint8:
return float64(x)
case uint16:
return float64(x)
case uint32:
return float64(x)
case uint64:
return float64(x)
default:
panic(fmt.Sprintf("invalid operation: float64(%T)", x))
}
}
func isNil(v interface{}) bool {
if v == nil {
return true
}
r := reflect.ValueOf(v)
switch r.Kind() {
case reflect.Chan, reflect.Func, reflect.Map, reflect.Ptr, reflect.Interface, reflect.Slice:
return r.IsNil()
default:
return false
}
}
+439
View File
@@ -0,0 +1,439 @@
package vm
import (
"fmt"
"reflect"
"regexp"
"strings"
"github.com/antonmedv/expr/file"
)
var (
MemoryBudget int = 1e6
)
func Run(program *Program, env interface{}) (interface{}, error) {
if program == nil {
return nil, fmt.Errorf("program is nil")
}
vm := VM{}
return vm.Run(program, env)
}
type VM struct {
stack []interface{}
constants []interface{}
bytecode []byte
ip int
pp int
scopes []Scope
debug bool
step chan struct{}
curr chan int
memory int
limit int
}
func Debug() *VM {
vm := &VM{
debug: true,
step: make(chan struct{}, 0),
curr: make(chan int, 0),
}
return vm
}
func (vm *VM) Run(program *Program, env interface{}) (out interface{}, err error) {
defer func() {
if r := recover(); r != nil {
f := &file.Error{
Location: program.Locations[vm.pp],
Message: fmt.Sprintf("%v", r),
}
err = f.Bind(program.Source)
}
}()
vm.limit = MemoryBudget
vm.ip = 0
vm.pp = 0
if vm.stack == nil {
vm.stack = make([]interface{}, 0, 2)
} else {
vm.stack = vm.stack[0:0]
}
if vm.scopes != nil {
vm.scopes = vm.scopes[0:0]
}
vm.bytecode = program.Bytecode
vm.constants = program.Constants
for vm.ip < len(vm.bytecode) {
if vm.debug {
<-vm.step
}
vm.pp = vm.ip
vm.ip++
op := vm.bytecode[vm.pp]
switch op {
case OpPush:
vm.push(vm.constant())
case OpPop:
vm.pop()
case OpRot:
b := vm.pop()
a := vm.pop()
vm.push(b)
vm.push(a)
case OpFetch:
vm.push(fetch(env, vm.constant()))
case OpFetchMap:
vm.push(env.(map[string]interface{})[vm.constant().(string)])
case OpTrue:
vm.push(true)
case OpFalse:
vm.push(false)
case OpNil:
vm.push(nil)
case OpNegate:
v := negate(vm.pop())
vm.push(v)
case OpNot:
v := vm.pop().(bool)
vm.push(!v)
case OpEqual:
b := vm.pop()
a := vm.pop()
vm.push(equal(a, b))
case OpEqualInt:
b := vm.pop()
a := vm.pop()
vm.push(a.(int) == b.(int))
case OpEqualString:
b := vm.pop()
a := vm.pop()
vm.push(a.(string) == b.(string))
case OpJump:
offset := vm.arg()
vm.ip += int(offset)
case OpJumpIfTrue:
offset := vm.arg()
if vm.current().(bool) {
vm.ip += int(offset)
}
case OpJumpIfFalse:
offset := vm.arg()
if !vm.current().(bool) {
vm.ip += int(offset)
}
case OpJumpBackward:
offset := vm.arg()
vm.ip -= int(offset)
case OpIn:
b := vm.pop()
a := vm.pop()
vm.push(in(a, b))
case OpLess:
b := vm.pop()
a := vm.pop()
vm.push(less(a, b))
case OpMore:
b := vm.pop()
a := vm.pop()
vm.push(more(a, b))
case OpLessOrEqual:
b := vm.pop()
a := vm.pop()
vm.push(lessOrEqual(a, b))
case OpMoreOrEqual:
b := vm.pop()
a := vm.pop()
vm.push(moreOrEqual(a, b))
case OpAdd:
b := vm.pop()
a := vm.pop()
vm.push(add(a, b))
case OpSubtract:
b := vm.pop()
a := vm.pop()
vm.push(subtract(a, b))
case OpMultiply:
b := vm.pop()
a := vm.pop()
vm.push(multiply(a, b))
case OpDivide:
b := vm.pop()
a := vm.pop()
vm.push(divide(a, b))
case OpModulo:
b := vm.pop()
a := vm.pop()
vm.push(modulo(a, b))
case OpExponent:
b := vm.pop()
a := vm.pop()
vm.push(exponent(a, b))
case OpRange:
b := vm.pop()
a := vm.pop()
min := toInt(a)
max := toInt(b)
size := max - min + 1
if vm.memory+size >= vm.limit {
panic("memory budget exceeded")
}
vm.push(makeRange(min, max))
vm.memory += size
case OpMatches:
b := vm.pop()
a := vm.pop()
match, err := regexp.MatchString(b.(string), a.(string))
if err != nil {
panic(err)
}
vm.push(match)
case OpMatchesConst:
a := vm.pop()
r := vm.constant().(*regexp.Regexp)
vm.push(r.MatchString(a.(string)))
case OpContains:
b := vm.pop()
a := vm.pop()
vm.push(strings.Contains(a.(string), b.(string)))
case OpStartsWith:
b := vm.pop()
a := vm.pop()
vm.push(strings.HasPrefix(a.(string), b.(string)))
case OpEndsWith:
b := vm.pop()
a := vm.pop()
vm.push(strings.HasSuffix(a.(string), b.(string)))
case OpIndex:
b := vm.pop()
a := vm.pop()
vm.push(fetch(a, b))
case OpSlice:
from := vm.pop()
to := vm.pop()
node := vm.pop()
vm.push(slice(node, from, to))
case OpProperty:
a := vm.pop()
b := vm.constant()
vm.push(fetch(a, b))
case OpCall:
call := vm.constant().(Call)
in := make([]reflect.Value, call.Size)
for i := call.Size - 1; i >= 0; i-- {
param := vm.pop()
if param == nil && reflect.TypeOf(param) == nil {
// In case of nil value and nil type use this hack,
// otherwise reflect.Call will panic on zero value.
in[i] = reflect.ValueOf(&param).Elem()
} else {
in[i] = reflect.ValueOf(param)
}
}
out := FetchFn(env, call.Name).Call(in)
vm.push(out[0].Interface())
case OpCallFast:
call := vm.constant().(Call)
in := make([]interface{}, call.Size)
for i := call.Size - 1; i >= 0; i-- {
in[i] = vm.pop()
}
fn := FetchFn(env, call.Name).Interface()
vm.push(fn.(func(...interface{}) interface{})(in...))
case OpMethod:
call := vm.constants[vm.arg()].(Call)
in := make([]reflect.Value, call.Size)
for i := call.Size - 1; i >= 0; i-- {
param := vm.pop()
if param == nil && reflect.TypeOf(param) == nil {
// In case of nil value and nil type use this hack,
// otherwise reflect.Call will panic on zero value.
in[i] = reflect.ValueOf(&param).Elem()
} else {
in[i] = reflect.ValueOf(param)
}
}
out := FetchFn(vm.pop(), call.Name).Call(in)
vm.push(out[0].Interface())
case OpArray:
size := vm.pop().(int)
array := make([]interface{}, size)
for i := size - 1; i >= 0; i-- {
array[i] = vm.pop()
}
vm.push(array)
vm.memory += size
if vm.memory >= vm.limit {
panic("memory budget exceeded")
}
case OpMap:
size := vm.pop().(int)
m := make(map[string]interface{})
for i := size - 1; i >= 0; i-- {
value := vm.pop()
key := vm.pop()
m[key.(string)] = value
}
vm.push(m)
vm.memory += size
if vm.memory >= vm.limit {
panic("memory budget exceeded")
}
case OpLen:
vm.push(length(vm.current()))
case OpCast:
t := vm.arg()
switch t {
case 0:
vm.push(toInt64(vm.pop()))
case 1:
vm.push(toFloat64(vm.pop()))
}
case OpStore:
scope := vm.Scope()
key := vm.constant().(string)
value := vm.pop()
scope[key] = value
case OpLoad:
scope := vm.Scope()
key := vm.constant().(string)
vm.push(scope[key])
case OpInc:
scope := vm.Scope()
key := vm.constant().(string)
i := scope[key].(int)
i++
scope[key] = i
case OpBegin:
scope := make(Scope)
vm.scopes = append(vm.scopes, scope)
case OpEnd:
vm.scopes = vm.scopes[:len(vm.scopes)-1]
default:
panic(fmt.Sprintf("unknown bytecode %#x", op))
}
if vm.debug {
vm.curr <- vm.ip
}
}
if vm.debug {
close(vm.curr)
close(vm.step)
}
if len(vm.stack) > 0 {
return vm.pop(), nil
}
return nil, nil
}
func (vm *VM) push(value interface{}) {
vm.stack = append(vm.stack, value)
}
func (vm *VM) current() interface{} {
return vm.stack[len(vm.stack)-1]
}
func (vm *VM) pop() interface{} {
value := vm.stack[len(vm.stack)-1]
vm.stack = vm.stack[:len(vm.stack)-1]
return value
}
func (vm *VM) arg() uint16 {
b0, b1 := vm.bytecode[vm.ip], vm.bytecode[vm.ip+1]
vm.ip += 2
return uint16(b0) | uint16(b1)<<8
}
func (vm *VM) constant() interface{} {
return vm.constants[vm.arg()]
}
func (vm *VM) Stack() []interface{} {
return vm.stack
}
func (vm *VM) Scope() Scope {
if len(vm.scopes) > 0 {
return vm.scopes[len(vm.scopes)-1]
}
return nil
}
func (vm *VM) Step() {
if vm.ip < len(vm.bytecode) {
vm.step <- struct{}{}
}
}
func (vm *VM) Position() chan int {
return vm.curr
}
+55
View File
@@ -0,0 +1,55 @@
Copyright (c) 2012-2016 Charles Banning <clbanning@gmail.com>. All rights reserved.
The MIT License (MIT)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
===============================================================================
Go Language Copyright & License -
Copyright 2009 The Go Authors. All rights reserved.
Use of this source code is governed by a BSD-style
license that can be found in the LICENSE file.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+189
View File
@@ -0,0 +1,189 @@
package mxj
import (
"encoding/xml"
"reflect"
)
const (
DefaultElementTag = "element"
)
// Encode arbitrary value as XML.
//
// Note: unmarshaling the resultant
// XML may not return the original value, since tag labels may have been injected
// to create the XML representation of the value.
/*
Encode an arbitrary JSON object.
package main
import (
"encoding/json"
"fmt"
"github.com/clbanning/mxj"
)
func main() {
jsondata := []byte(`[
{ "somekey":"somevalue" },
"string",
3.14159265,
true
]`)
var i interface{}
err := json.Unmarshal(jsondata, &i)
if err != nil {
// do something
}
x, err := mxj.AnyXmlIndent(i, "", " ", "mydoc")
if err != nil {
// do something else
}
fmt.Println(string(x))
}
output:
<mydoc>
<somekey>somevalue</somekey>
<element>string</element>
<element>3.14159265</element>
<element>true</element>
</mydoc>
*/
// Alternative values for DefaultRootTag and DefaultElementTag can be set as:
// AnyXml( v, myRootTag, myElementTag).
func AnyXml(v interface{}, tags ...string) ([]byte, error) {
var rt, et string
if len(tags) == 1 || len(tags) == 2 {
rt = tags[0]
} else {
rt = DefaultRootTag
}
if len(tags) == 2 {
et = tags[1]
} else {
et = DefaultElementTag
}
if v == nil {
if useGoXmlEmptyElemSyntax {
return []byte("<" + rt + "></" + rt + ">"), nil
}
return []byte("<" + rt + "/>"), nil
}
if reflect.TypeOf(v).Kind() == reflect.Struct {
return xml.Marshal(v)
}
var err error
s := new(string)
p := new(pretty)
var ss string
var b []byte
switch v.(type) {
case []interface{}:
ss = "<" + rt + ">"
for _, vv := range v.([]interface{}) {
switch vv.(type) {
case map[string]interface{}:
m := vv.(map[string]interface{})
if len(m) == 1 {
for tag, val := range m {
err = mapToXmlIndent(false, s, tag, val, p)
}
} else {
err = mapToXmlIndent(false, s, et, vv, p)
}
default:
err = mapToXmlIndent(false, s, et, vv, p)
}
if err != nil {
break
}
}
ss += *s + "</" + rt + ">"
b = []byte(ss)
case map[string]interface{}:
m := Map(v.(map[string]interface{}))
b, err = m.Xml(rt)
default:
err = mapToXmlIndent(false, s, rt, v, p)
b = []byte(*s)
}
return b, err
}
// Encode an arbitrary value as a pretty XML string.
// Alternative values for DefaultRootTag and DefaultElementTag can be set as:
// AnyXmlIndent( v, "", " ", myRootTag, myElementTag).
func AnyXmlIndent(v interface{}, prefix, indent string, tags ...string) ([]byte, error) {
var rt, et string
if len(tags) == 1 || len(tags) == 2 {
rt = tags[0]
} else {
rt = DefaultRootTag
}
if len(tags) == 2 {
et = tags[1]
} else {
et = DefaultElementTag
}
if v == nil {
if useGoXmlEmptyElemSyntax {
return []byte(prefix + "<" + rt + "></" + rt + ">"), nil
}
return []byte(prefix + "<" + rt + "/>"), nil
}
if reflect.TypeOf(v).Kind() == reflect.Struct {
return xml.MarshalIndent(v, prefix, indent)
}
var err error
s := new(string)
p := new(pretty)
p.indent = indent
p.padding = prefix
var ss string
var b []byte
switch v.(type) {
case []interface{}:
ss = "<" + rt + ">\n"
p.Indent()
for _, vv := range v.([]interface{}) {
switch vv.(type) {
case map[string]interface{}:
m := vv.(map[string]interface{})
if len(m) == 1 {
for tag, val := range m {
err = mapToXmlIndent(true, s, tag, val, p)
}
} else {
p.start = 1 // we 1 tag in
err = mapToXmlIndent(true, s, et, vv, p)
*s += "\n"
}
default:
p.start = 0 // in case trailing p.start = 1
err = mapToXmlIndent(true, s, et, vv, p)
}
if err != nil {
break
}
}
ss += *s + "</" + rt + ">"
b = []byte(ss)
case map[string]interface{}:
m := Map(v.(map[string]interface{}))
b, err = m.XmlIndent(prefix, indent, rt)
default:
err = mapToXmlIndent(true, s, rt, v, p)
b = []byte(*s)
}
return b, err
}
+54
View File
@@ -0,0 +1,54 @@
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-us" updated="2009-10-04T01:35:58+00:00"><title>Code Review - My issues</title><link href="http://codereview.appspot.com/" rel="alternate"></link><link href="http://codereview.appspot.com/rss/mine/rsc" rel="self"></link><id>http://codereview.appspot.com/</id><author><name>rietveld&lt;&gt;</name></author><entry><title>rietveld: an attempt at pubsubhubbub
</title><link href="http://codereview.appspot.com/126085" rel="alternate"></link><updated>2009-10-04T01:35:58+00:00</updated><author><name>email-address-removed</name></author><id>urn:md5:134d9179c41f806be79b3a5f7877d19a</id><summary type="html">
An attempt at adding pubsubhubbub support to Rietveld.
http://code.google.com/p/pubsubhubbub
http://code.google.com/p/rietveld/issues/detail?id=155
The server side of the protocol is trivial:
1. add a &amp;lt;link rel=&amp;quot;hub&amp;quot; href=&amp;quot;hub-server&amp;quot;&amp;gt; tag to all
feeds that will be pubsubhubbubbed.
2. every time one of those feeds changes, tell the hub
with a simple POST request.
I have tested this by adding debug prints to a local hub
server and checking that the server got the right publish
requests.
I can&amp;#39;t quite get the server to work, but I think the bug
is not in my code. I think that the server expects to be
able to grab the feed and see the feed&amp;#39;s actual URL in
the link rel=&amp;quot;self&amp;quot;, but the default value for that drops
the :port from the URL, and I cannot for the life of me
figure out how to get the Atom generator deep inside
django not to do that, or even where it is doing that,
or even what code is running to generate the Atom feed.
(I thought I knew but I added some assert False statements
and it kept running!)
Ignoring that particular problem, I would appreciate
feedback on the right way to get the two values at
the top of feeds.py marked NOTE(rsc).
</summary></entry><entry><title>rietveld: correct tab handling
</title><link href="http://codereview.appspot.com/124106" rel="alternate"></link><updated>2009-10-03T23:02:17+00:00</updated><author><name>email-address-removed</name></author><id>urn:md5:0a2a4f19bb815101f0ba2904aed7c35a</id><summary type="html">
This fixes the buggy tab rendering that can be seen at
http://codereview.appspot.com/116075/diff/1/2
The fundamental problem was that the tab code was
not being told what column the text began in, so it
didn&amp;#39;t know where to put the tab stops. Another problem
was that some of the code assumed that string byte
offsets were the same as column offsets, which is only
true if there are no tabs.
In the process of fixing this, I cleaned up the arguments
to Fold and ExpandTabs and renamed them Break and
_ExpandTabs so that I could be sure that I found all the
call sites. I also wanted to verify that ExpandTabs was
not being used from outside intra_region_diff.py.
</summary></entry></feed> `
+134
View File
@@ -0,0 +1,134 @@
// mxj - A collection of map[string]interface{} and associated XML and JSON utilities.
// Copyright 2012-2015, 2018 Charles Banning. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file
/*
Marshal/Unmarshal XML to/from map[string]interface{} values (and JSON); extract/modify values from maps by key or key-path, including wildcards.
mxj supplants the legacy x2j and j2x packages. The subpackage x2j-wrapper is provided to facilitate migrating from the x2j package. The x2j and j2x subpackages provide similar functionality of the old packages but are not function-name compatible with them.
Note: this library was designed for processing ad hoc anonymous messages. Bulk processing large data sets may be much more efficiently performed using the encoding/xml or encoding/json packages from Go's standard library directly.
Related Packages:
checkxml: github.com/clbanning/checkxml provides functions for validating XML data.
Notes:
2018.04.18: mv.Xml/mv.XmlIndent encodes non-map[string]interface{} map values - map[string]string, map[int]uint, etc.
2018.03.29: mv.Gob/NewMapGob support gob encoding/decoding of Maps.
2018.03.26: Added mxj/x2j-wrapper sub-package for migrating from legacy x2j package.
2017.02.22: LeafNode paths can use ".N" syntax rather than "[N]" for list member indexing.
2017.02.21: github.com/clbanning/checkxml provides functions for validating XML data.
2017.02.10: SetFieldSeparator changes field separator for args in UpdateValuesForPath, ValuesFor... methods.
2017.02.06: Support XMPP stream processing - HandleXMPPStreamTag().
2016.11.07: Preserve name space prefix syntax in XmlSeq parser - NewMapXmlSeq(), etc.
2016.06.25: Support overriding default XML attribute prefix, "-", in Map keys - SetAttrPrefix().
2016.05.26: Support customization of xml.Decoder by exposing CustomDecoder variable.
2016.03.19: Escape invalid chars when encoding XML attribute and element values - XMLEscapeChars().
2016.03.02: By default decoding XML with float64 and bool value casting will not cast "NaN", "Inf", and "-Inf".
To cast them to float64, first set flag with CastNanInf(true).
2016.02.22: New mv.Root(), mv.Elements(), mv.Attributes methods let you examine XML document structure.
2016.02.16: Add CoerceKeysToLower() option to handle tags with mixed capitalization.
2016.02.12: Seek for first xml.StartElement token; only return error if io.EOF is reached first (handles BOM).
2015-12-02: NewMapXmlSeq() with mv.XmlSeq() & co. will try to preserve structure of XML doc when re-encoding.
2014-08-02: AnyXml() and AnyXmlIndent() will try to marshal arbitrary values to XML.
SUMMARY
type Map map[string]interface{}
Create a Map value, 'mv', from any map[string]interface{} value, 'v':
mv := Map(v)
Unmarshal / marshal XML as a Map value, 'mv':
mv, err := NewMapXml(xmlValue) // unmarshal
xmlValue, err := mv.Xml() // marshal
Unmarshal XML from an io.Reader as a Map value, 'mv':
mv, err := NewMapXmlReader(xmlReader) // repeated calls, as with an os.File Reader, will process stream
mv, raw, err := NewMapXmlReaderRaw(xmlReader) // 'raw' is the raw XML that was decoded
Marshal Map value, 'mv', to an XML Writer (io.Writer):
err := mv.XmlWriter(xmlWriter)
raw, err := mv.XmlWriterRaw(xmlWriter) // 'raw' is the raw XML that was written on xmlWriter
Also, for prettified output:
xmlValue, err := mv.XmlIndent(prefix, indent, ...)
err := mv.XmlIndentWriter(xmlWriter, prefix, indent, ...)
raw, err := mv.XmlIndentWriterRaw(xmlWriter, prefix, indent, ...)
Bulk process XML with error handling (note: handlers must return a boolean value):
err := HandleXmlReader(xmlReader, mapHandler(Map), errHandler(error))
err := HandleXmlReaderRaw(xmlReader, mapHandler(Map, []byte), errHandler(error, []byte))
Converting XML to JSON: see Examples for NewMapXml and HandleXmlReader.
There are comparable functions and methods for JSON processing.
Arbitrary structure values can be decoded to / encoded from Map values:
mv, err := NewMapStruct(structVal)
err := mv.Struct(structPointer)
To work with XML tag values, JSON or Map key values or structure field values, decode the XML, JSON
or structure to a Map value, 'mv', or cast a map[string]interface{} value to a Map value, 'mv', then:
paths := mv.PathsForKey(key)
path := mv.PathForKeyShortest(key)
values, err := mv.ValuesForKey(key, subkeys)
values, err := mv.ValuesForPath(path, subkeys) // 'path' can be dot-notation with wildcards and indexed arrays.
count, err := mv.UpdateValuesForPath(newVal, path, subkeys)
Get everything at once, irrespective of path depth:
leafnodes := mv.LeafNodes()
leafvalues := mv.LeafValues()
A new Map with whatever keys are desired can be created from the current Map and then encoded in XML
or JSON. (Note: keys can use dot-notation. 'oldKey' can also use wildcards and indexed arrays.)
newMap, err := mv.NewMap("oldKey_1:newKey_1", "oldKey_2:newKey_2", ..., "oldKey_N:newKey_N")
newMap, err := mv.NewMap("oldKey1", "oldKey3", "oldKey5") // a subset of 'mv'; see "examples/partial.go"
newXml, err := newMap.Xml() // for example
newJson, err := newMap.Json() // ditto
XML PARSING CONVENTIONS
Using NewMapXml()
- Attributes are parsed to `map[string]interface{}` values by prefixing a hyphen, `-`,
to the attribute label. (Unless overridden by `PrependAttrWithHyphen(false)` or
`SetAttrPrefix()`.)
- If the element is a simple element and has attributes, the element value
is given the key `#text` for its `map[string]interface{}` representation. (See
the 'atomFeedString.xml' test data, below.)
- XML comments, directives, and process instructions are ignored.
- If CoerceKeysToLower() has been called, then the resultant keys will be lower case.
Using NewMapXmlSeq()
- Attributes are parsed to `map["#attr"]map[<attr_label>]map[string]interface{}`values
where the `<attr_label>` value has "#text" and "#seq" keys - the "#text" key holds the
value for `<attr_label>`.
- All elements, except for the root, have a "#seq" key.
- Comments, directives, and process instructions are unmarshalled into the Map using the
keys "#comment", "#directive", and "#procinst", respectively. (See documentation for more
specifics.)
- Name space syntax is preserved:
- <ns:key>something</ns.key> parses to map["ns:key"]interface{}{"something"}
- xmlns:ns="http://myns.com/ns" parses to map["xmlns:ns"]interface{}{"http://myns.com/ns"}
Both
- By default, "Nan", "Inf", and "-Inf" values are not cast to float64. If you want them
to be cast, set a flag to cast them using CastNanInf(true).
XML ENCODING CONVENTIONS
- 'nil' Map values, which may represent 'null' JSON values, are encoded as "<tag/>".
NOTE: the operation is not symmetric as "<tag/>" elements are decoded as 'tag:""' Map values,
which, then, encode in JSON as '"tag":""' values..
- ALSO: there is no guarantee that the encoded XML doc will be the same as the decoded one. (Go
randomizes the walk through map[string]interface{} values.) If you plan to re-encode the
Map value to XML and want the same sequencing of elements look at NewMapXmlSeq() and
mv.XmlSeq() - these try to preserve the element sequencing but with added complexity when
working with the Map representation.
*/
package mxj
+54
View File
@@ -0,0 +1,54 @@
// Copyright 2016 Charles Banning. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file
package mxj
import (
"bytes"
)
var xmlEscapeChars bool
// XMLEscapeChars(true) forces escaping invalid characters in attribute and element values.
// NOTE: this is brute force with NO interrogation of '&' being escaped already; if it is
// then '&amp;' will be re-escaped as '&amp;amp;'.
//
/*
The values are:
" &quot;
' &apos;
< &lt;
> &gt;
& &amp;
*/
func XMLEscapeChars(b bool) {
xmlEscapeChars = b
}
// Scan for '&' first, since 's' may contain "&amp;" that is parsed to "&amp;amp;"
// - or "&lt;" that is parsed to "&amp;lt;".
var escapechars = [][2][]byte{
{[]byte(`&`), []byte(`&amp;`)},
{[]byte(`<`), []byte(`&lt;`)},
{[]byte(`>`), []byte(`&gt;`)},
{[]byte(`"`), []byte(`&quot;`)},
{[]byte(`'`), []byte(`&apos;`)},
}
func escapeChars(s string) string {
if len(s) == 0 {
return s
}
b := []byte(s)
for _, v := range escapechars {
n := bytes.Count(b, v[0])
if n == 0 {
continue
}
b = bytes.Replace(b, v[0], v[1], n)
}
return string(b)
}
+7
View File
@@ -0,0 +1,7 @@
package mxj
// Checks whether the path exists
func (mv Map) Exists(path string, subkeys ...string) bool {
v, err := mv.ValuesForPath(path, subkeys...)
return err == nil && len(v) > 0
}
+287
View File
@@ -0,0 +1,287 @@
package mxj
import (
"fmt"
"io"
"os"
)
type Maps []Map
func NewMaps() Maps {
return make(Maps, 0)
}
type MapRaw struct {
M Map
R []byte
}
// NewMapsFromXmlFile - creates an array from a file of JSON values.
func NewMapsFromJsonFile(name string) (Maps, error) {
fi, err := os.Stat(name)
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("file %s is not a regular file", name)
}
fh, err := os.Open(name)
if err != nil {
return nil, err
}
defer fh.Close()
am := make([]Map, 0)
for {
m, raw, err := NewMapJsonReaderRaw(fh)
if err != nil && err != io.EOF {
return am, fmt.Errorf("error: %s - reading: %s", err.Error(), string(raw))
}
if len(m) > 0 {
am = append(am, m)
}
if err == io.EOF {
break
}
}
return am, nil
}
// ReadMapsFromJsonFileRaw - creates an array of MapRaw from a file of JSON values.
func NewMapsFromJsonFileRaw(name string) ([]MapRaw, error) {
fi, err := os.Stat(name)
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("file %s is not a regular file", name)
}
fh, err := os.Open(name)
if err != nil {
return nil, err
}
defer fh.Close()
am := make([]MapRaw, 0)
for {
mr := new(MapRaw)
mr.M, mr.R, err = NewMapJsonReaderRaw(fh)
if err != nil && err != io.EOF {
return am, fmt.Errorf("error: %s - reading: %s", err.Error(), string(mr.R))
}
if len(mr.M) > 0 {
am = append(am, *mr)
}
if err == io.EOF {
break
}
}
return am, nil
}
// NewMapsFromXmlFile - creates an array from a file of XML values.
func NewMapsFromXmlFile(name string) (Maps, error) {
fi, err := os.Stat(name)
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("file %s is not a regular file", name)
}
fh, err := os.Open(name)
if err != nil {
return nil, err
}
defer fh.Close()
am := make([]Map, 0)
for {
m, raw, err := NewMapXmlReaderRaw(fh)
if err != nil && err != io.EOF {
return am, fmt.Errorf("error: %s - reading: %s", err.Error(), string(raw))
}
if len(m) > 0 {
am = append(am, m)
}
if err == io.EOF {
break
}
}
return am, nil
}
// NewMapsFromXmlFileRaw - creates an array of MapRaw from a file of XML values.
// NOTE: the slice with the raw XML is clean with no extra capacity - unlike NewMapXmlReaderRaw().
// It is slow at parsing a file from disk and is intended for relatively small utility files.
func NewMapsFromXmlFileRaw(name string) ([]MapRaw, error) {
fi, err := os.Stat(name)
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("file %s is not a regular file", name)
}
fh, err := os.Open(name)
if err != nil {
return nil, err
}
defer fh.Close()
am := make([]MapRaw, 0)
for {
mr := new(MapRaw)
mr.M, mr.R, err = NewMapXmlReaderRaw(fh)
if err != nil && err != io.EOF {
return am, fmt.Errorf("error: %s - reading: %s", err.Error(), string(mr.R))
}
if len(mr.M) > 0 {
am = append(am, *mr)
}
if err == io.EOF {
break
}
}
return am, nil
}
// ------------------------ Maps writing -------------------------
// These are handy-dandy methods for dumping configuration data, etc.
// JsonString - analogous to mv.Json()
func (mvs Maps) JsonString(safeEncoding ...bool) (string, error) {
var s string
for _, v := range mvs {
j, err := v.Json()
if err != nil {
return s, err
}
s += string(j)
}
return s, nil
}
// JsonStringIndent - analogous to mv.JsonIndent()
func (mvs Maps) JsonStringIndent(prefix, indent string, safeEncoding ...bool) (string, error) {
var s string
var haveFirst bool
for _, v := range mvs {
j, err := v.JsonIndent(prefix, indent)
if err != nil {
return s, err
}
if haveFirst {
s += "\n"
} else {
haveFirst = true
}
s += string(j)
}
return s, nil
}
// XmlString - analogous to mv.Xml()
func (mvs Maps) XmlString() (string, error) {
var s string
for _, v := range mvs {
x, err := v.Xml()
if err != nil {
return s, err
}
s += string(x)
}
return s, nil
}
// XmlStringIndent - analogous to mv.XmlIndent()
func (mvs Maps) XmlStringIndent(prefix, indent string) (string, error) {
var s string
for _, v := range mvs {
x, err := v.XmlIndent(prefix, indent)
if err != nil {
return s, err
}
s += string(x)
}
return s, nil
}
// JsonFile - write Maps to named file as JSON
// Note: the file will be created, if necessary; if it exists it will be truncated.
// If you need to append to a file, open it and use JsonWriter method.
func (mvs Maps) JsonFile(file string, safeEncoding ...bool) error {
var encoding bool
if len(safeEncoding) == 1 {
encoding = safeEncoding[0]
}
s, err := mvs.JsonString(encoding)
if err != nil {
return err
}
fh, err := os.Create(file)
if err != nil {
return err
}
defer fh.Close()
fh.WriteString(s)
return nil
}
// JsonFileIndent - write Maps to named file as pretty JSON
// Note: the file will be created, if necessary; if it exists it will be truncated.
// If you need to append to a file, open it and use JsonIndentWriter method.
func (mvs Maps) JsonFileIndent(file, prefix, indent string, safeEncoding ...bool) error {
var encoding bool
if len(safeEncoding) == 1 {
encoding = safeEncoding[0]
}
s, err := mvs.JsonStringIndent(prefix, indent, encoding)
if err != nil {
return err
}
fh, err := os.Create(file)
if err != nil {
return err
}
defer fh.Close()
fh.WriteString(s)
return nil
}
// XmlFile - write Maps to named file as XML
// Note: the file will be created, if necessary; if it exists it will be truncated.
// If you need to append to a file, open it and use XmlWriter method.
func (mvs Maps) XmlFile(file string) error {
s, err := mvs.XmlString()
if err != nil {
return err
}
fh, err := os.Create(file)
if err != nil {
return err
}
defer fh.Close()
fh.WriteString(s)
return nil
}
// XmlFileIndent - write Maps to named file as pretty XML
// Note: the file will be created,if necessary; if it exists it will be truncated.
// If you need to append to a file, open it and use XmlIndentWriter method.
func (mvs Maps) XmlFileIndent(file, prefix, indent string) error {
s, err := mvs.XmlStringIndent(prefix, indent)
if err != nil {
return err
}
fh, err := os.Create(file)
if err != nil {
return err
}
defer fh.Close()
fh.WriteString(s)
return nil
}
+2
View File
@@ -0,0 +1,2 @@
{ "this":"is", "a":"test", "file":"for", "files_test.go":"case" }
{ "with":"some", "bad":JSON, "in":"it" }
+9
View File
@@ -0,0 +1,9 @@
<doc>
<some>test</some>
<data>for files.go</data>
</doc>
<msg>
<just>some</just>
<another>doc</other>
<for>test case</for>
</msg>
+2
View File
@@ -0,0 +1,2 @@
{ "this":"is", "a":"test", "file":"for", "files_test.go":"case" }
{ "with":"just", "two":2, "JSON":"values", "true":true }
+9
View File
@@ -0,0 +1,9 @@
<doc>
<some>test</some>
<data>for files.go</data>
</doc>
<msg>
<just>some</just>
<another>doc</another>
<for>test case</for>
</msg>
+1
View File
@@ -0,0 +1 @@
{"a":"test","file":"for","files_test.go":"case","this":"is"}{"JSON":"values","true":true,"two":2,"with":"just"}
+1
View File
@@ -0,0 +1 @@
<doc><data>for files.go</data><some>test</some></doc><msg><another>doc</another><for>test case</for><just>some</just></msg>
+12
View File
@@ -0,0 +1,12 @@
{
"a": "test",
"file": "for",
"files_test.go": "case",
"this": "is"
}
{
"JSON": "values",
"true": true,
"two": 2,
"with": "just"
}
+8
View File
@@ -0,0 +1,8 @@
<doc>
<data>for files.go</data>
<some>test</some>
</doc><msg>
<another>doc</another>
<for>test case</for>
<just>some</just>
</msg>
+35
View File
@@ -0,0 +1,35 @@
// gob.go - Encode/Decode a Map into a gob object.
package mxj
import (
"bytes"
"encoding/gob"
)
// NewMapGob returns a Map value for a gob object that has been
// encoded from a map[string]interface{} (or compatible type) value.
// It is intended to provide symmetric handling of Maps that have
// been encoded using mv.Gob.
func NewMapGob(gobj []byte) (Map, error) {
m := make(map[string]interface{}, 0)
if len(gobj) == 0 {
return m, nil
}
r := bytes.NewReader(gobj)
dec := gob.NewDecoder(r)
if err := dec.Decode(&m); err != nil {
return m, err
}
return m, nil
}
// Gob returns a gob-encoded value for the Map 'mv'.
func (mv Map) Gob() ([]byte, error) {
var buf bytes.Buffer
enc := gob.NewEncoder(&buf)
if err := enc.Encode(map[string]interface{}(mv)); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
+323
View File
@@ -0,0 +1,323 @@
// Copyright 2012-2014 Charles Banning. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file
package mxj
import (
"bytes"
"encoding/json"
"fmt"
"io"
"time"
)
// ------------------------------ write JSON -----------------------
// Just a wrapper on json.Marshal.
// If option safeEncoding is'true' then safe encoding of '<', '>' and '&'
// is preserved. (see encoding/json#Marshal, encoding/json#Encode)
func (mv Map) Json(safeEncoding ...bool) ([]byte, error) {
var s bool
if len(safeEncoding) == 1 {
s = safeEncoding[0]
}
b, err := json.Marshal(mv)
if !s {
b = bytes.Replace(b, []byte("\\u003c"), []byte("<"), -1)
b = bytes.Replace(b, []byte("\\u003e"), []byte(">"), -1)
b = bytes.Replace(b, []byte("\\u0026"), []byte("&"), -1)
}
return b, err
}
// Just a wrapper on json.MarshalIndent.
// If option safeEncoding is'true' then safe encoding of '<' , '>' and '&'
// is preserved. (see encoding/json#Marshal, encoding/json#Encode)
func (mv Map) JsonIndent(prefix, indent string, safeEncoding ...bool) ([]byte, error) {
var s bool
if len(safeEncoding) == 1 {
s = safeEncoding[0]
}
b, err := json.MarshalIndent(mv, prefix, indent)
if !s {
b = bytes.Replace(b, []byte("\\u003c"), []byte("<"), -1)
b = bytes.Replace(b, []byte("\\u003e"), []byte(">"), -1)
b = bytes.Replace(b, []byte("\\u0026"), []byte("&"), -1)
}
return b, err
}
// The following implementation is provided for symmetry with NewMapJsonReader[Raw]
// The names will also provide a key for the number of return arguments.
// Writes the Map as JSON on the Writer.
// If 'safeEncoding' is 'true', then "safe" encoding of '<', '>' and '&' is preserved.
func (mv Map) JsonWriter(jsonWriter io.Writer, safeEncoding ...bool) error {
b, err := mv.Json(safeEncoding...)
if err != nil {
return err
}
_, err = jsonWriter.Write(b)
return err
}
// Writes the Map as JSON on the Writer. []byte is the raw JSON that was written.
// If 'safeEncoding' is 'true', then "safe" encoding of '<', '>' and '&' is preserved.
func (mv Map) JsonWriterRaw(jsonWriter io.Writer, safeEncoding ...bool) ([]byte, error) {
b, err := mv.Json(safeEncoding...)
if err != nil {
return b, err
}
_, err = jsonWriter.Write(b)
return b, err
}
// Writes the Map as pretty JSON on the Writer.
// If 'safeEncoding' is 'true', then "safe" encoding of '<', '>' and '&' is preserved.
func (mv Map) JsonIndentWriter(jsonWriter io.Writer, prefix, indent string, safeEncoding ...bool) error {
b, err := mv.JsonIndent(prefix, indent, safeEncoding...)
if err != nil {
return err
}
_, err = jsonWriter.Write(b)
return err
}
// Writes the Map as pretty JSON on the Writer. []byte is the raw JSON that was written.
// If 'safeEncoding' is 'true', then "safe" encoding of '<', '>' and '&' is preserved.
func (mv Map) JsonIndentWriterRaw(jsonWriter io.Writer, prefix, indent string, safeEncoding ...bool) ([]byte, error) {
b, err := mv.JsonIndent(prefix, indent, safeEncoding...)
if err != nil {
return b, err
}
_, err = jsonWriter.Write(b)
return b, err
}
// --------------------------- read JSON -----------------------------
// Decode numericvalues as json.Number type Map values - see encoding/json#Number.
// NOTE: this is for decoding JSON into a Map with NewMapJson(), NewMapJsonReader(),
// etc.; it does not affect NewMapXml(), etc. The XML encoders mv.Xml() and mv.XmlIndent()
// do recognize json.Number types; a JSON object can be decoded to a Map with json.Number
// value types and the resulting Map can be correctly encoded into a XML object.
var JsonUseNumber bool
// Just a wrapper on json.Unmarshal
// Converting JSON to XML is a simple as:
// ...
// mapVal, merr := mxj.NewMapJson(jsonVal)
// if merr != nil {
// // handle error
// }
// xmlVal, xerr := mapVal.Xml()
// if xerr != nil {
// // handle error
// }
// NOTE: as a special case, passing a list, e.g., [{"some-null-value":"", "a-non-null-value":"bar"}],
// will be interpreted as having the root key 'object' prepended - {"object":[ ... ]} - to unmarshal to a Map.
// See mxj/j2x/j2x_test.go.
func NewMapJson(jsonVal []byte) (Map, error) {
// empty or nil begets empty
if len(jsonVal) == 0 {
m := make(map[string]interface{}, 0)
return m, nil
}
// handle a goofy case ...
if jsonVal[0] == '[' {
jsonVal = []byte(`{"object":` + string(jsonVal) + `}`)
}
m := make(map[string]interface{})
// err := json.Unmarshal(jsonVal, &m)
buf := bytes.NewReader(jsonVal)
dec := json.NewDecoder(buf)
if JsonUseNumber {
dec.UseNumber()
}
err := dec.Decode(&m)
return m, err
}
// Retrieve a Map value from an io.Reader.
// NOTE: The raw JSON off the reader is buffered to []byte using a ByteReader. If the io.Reader is an
// os.File, there may be significant performance impact. If the io.Reader is wrapping a []byte
// value in-memory, however, such as http.Request.Body you CAN use it to efficiently unmarshal
// a JSON object.
func NewMapJsonReader(jsonReader io.Reader) (Map, error) {
jb, err := getJson(jsonReader)
if err != nil || len(*jb) == 0 {
return nil, err
}
// Unmarshal the 'presumed' JSON string
return NewMapJson(*jb)
}
// Retrieve a Map value and raw JSON - []byte - from an io.Reader.
// NOTE: The raw JSON off the reader is buffered to []byte using a ByteReader. If the io.Reader is an
// os.File, there may be significant performance impact. If the io.Reader is wrapping a []byte
// value in-memory, however, such as http.Request.Body you CAN use it to efficiently unmarshal
// a JSON object and retrieve the raw JSON in a single call.
func NewMapJsonReaderRaw(jsonReader io.Reader) (Map, []byte, error) {
jb, err := getJson(jsonReader)
if err != nil || len(*jb) == 0 {
return nil, *jb, err
}
// Unmarshal the 'presumed' JSON string
m, merr := NewMapJson(*jb)
return m, *jb, merr
}
// Pull the next JSON string off the stream: just read from first '{' to its closing '}'.
// Returning a pointer to the slice saves 16 bytes - maybe unnecessary, but internal to package.
func getJson(rdr io.Reader) (*[]byte, error) {
bval := make([]byte, 1)
jb := make([]byte, 0)
var inQuote, inJson bool
var parenCnt int
var previous byte
// scan the input for a matched set of {...}
// json.Unmarshal will handle syntax checking.
for {
_, err := rdr.Read(bval)
if err != nil {
if err == io.EOF && inJson && parenCnt > 0 {
return &jb, fmt.Errorf("no closing } for JSON string: %s", string(jb))
}
return &jb, err
}
switch bval[0] {
case '{':
if !inQuote {
parenCnt++
inJson = true
}
case '}':
if !inQuote {
parenCnt--
}
if parenCnt < 0 {
return nil, fmt.Errorf("closing } without opening {: %s", string(jb))
}
case '"':
if inQuote {
if previous == '\\' {
break
}
inQuote = false
} else {
inQuote = true
}
case '\n', '\r', '\t', ' ':
if !inQuote {
continue
}
}
if inJson {
jb = append(jb, bval[0])
if parenCnt == 0 {
break
}
}
previous = bval[0]
}
return &jb, nil
}
// ------------------------------- JSON Reader handler via Map values -----------------------
// Default poll delay to keep Handler from spinning on an open stream
// like sitting on os.Stdin waiting for imput.
var jhandlerPollInterval = time.Duration(1e6)
// While unnecessary, we make HandleJsonReader() have the same signature as HandleXmlReader().
// This avoids treating one or other as a special case and discussing the underlying stdlib logic.
// Bulk process JSON using handlers that process a Map value.
// 'rdr' is an io.Reader for the JSON (stream).
// 'mapHandler' is the Map processing handler. Return of 'false' stops io.Reader processing.
// 'errHandler' is the error processor. Return of 'false' stops io.Reader processing and returns the error.
// Note: mapHandler() and errHandler() calls are blocking, so reading and processing of messages is serialized.
// This means that you can stop reading the file on error or after processing a particular message.
// To have reading and handling run concurrently, pass argument to a go routine in handler and return 'true'.
func HandleJsonReader(jsonReader io.Reader, mapHandler func(Map) bool, errHandler func(error) bool) error {
var n int
for {
m, merr := NewMapJsonReader(jsonReader)
n++
// handle error condition with errhandler
if merr != nil && merr != io.EOF {
merr = fmt.Errorf("[jsonReader: %d] %s", n, merr.Error())
if ok := errHandler(merr); !ok {
// caused reader termination
return merr
}
continue
}
// pass to maphandler
if len(m) != 0 {
if ok := mapHandler(m); !ok {
break
}
} else if merr != io.EOF {
<-time.After(jhandlerPollInterval)
}
if merr == io.EOF {
break
}
}
return nil
}
// Bulk process JSON using handlers that process a Map value and the raw JSON.
// 'rdr' is an io.Reader for the JSON (stream).
// 'mapHandler' is the Map and raw JSON - []byte - processor. Return of 'false' stops io.Reader processing.
// 'errHandler' is the error and raw JSON processor. Return of 'false' stops io.Reader processing and returns the error.
// Note: mapHandler() and errHandler() calls are blocking, so reading and processing of messages is serialized.
// This means that you can stop reading the file on error or after processing a particular message.
// To have reading and handling run concurrently, pass argument(s) to a go routine in handler and return 'true'.
func HandleJsonReaderRaw(jsonReader io.Reader, mapHandler func(Map, []byte) bool, errHandler func(error, []byte) bool) error {
var n int
for {
m, raw, merr := NewMapJsonReaderRaw(jsonReader)
n++
// handle error condition with errhandler
if merr != nil && merr != io.EOF {
merr = fmt.Errorf("[jsonReader: %d] %s", n, merr.Error())
if ok := errHandler(merr, raw); !ok {
// caused reader termination
return merr
}
continue
}
// pass to maphandler
if len(m) != 0 {
if ok := mapHandler(m, raw); !ok {
break
}
} else if merr != io.EOF {
<-time.After(jhandlerPollInterval)
}
if merr == io.EOF {
break
}
}
return nil
}

Some files were not shown because too many files have changed in this diff Show More