Merge pull request #250 from cheshirekow/cheshirekow-jira

Implement jira bridge

1 files changed, 377 insertions, 0 deletions

diff --git a/doc/jira_bridge.md b/doc/jira_bridge.md
new file mode 100644
index 00000000..df56bb2d
--- /dev/null
+++ b/doc/jira_bridge.md
@@ -0,0 +1,377 @@
+# JIRA Bridge
+
+## Design Notes
+
+### One bridge = one project
+
+There aren't any huge technical barriers requiring this, but since git-bug lacks
+a notion of "project" there is no way to know which project to export new bugs
+to as issues. Also, JIRA projects are first-class immutable metadata and so we
+*must* get it right on export. Therefore the bridge is configured with the `Key`
+for the project it is assigned to. It will only import bugs from that project.
+
+### JIRA fields
+
+The bridge currently does nothing to import any of the JIRA fields that don't
+have `git-bug` equivalents ("Assignee", "sprint", "story points", etc).
+Hopefully the bridge will be able to enable synchronization of these soon.
+
+### Credentials
+
+JIRA does not support user/personal access tokens. They have experimental
+3-legged oauth support but that requires an API token for the app configured
+by the server administrator. The only reliable authentication mechanism then is
+the username/password and session-token mechanims. We can aquire a session
+token programatically from the username/password but these are very short lived
+(i.e. hours or less). As such the bridge currently requires an actual username
+and password as user credentials. It supports three options:
+
+1. Storing both username and password in a separate file referred to by
+   the `git-config` (I like to use `.git/jira-credentials.json`)
+2. Storing the username and password in clear-text in the git config
+3. Storing the username only in the git config and asking for the password
+   on each `push` or `pull`.
+
+### Issue Creation Defaults
+
+When a new issues is created in JIRA there are often certain mandatory fields
+that require a value or the creation is rejected. In the issue create form on
+the JIRA web interface, these are annotated as "required". The `issuetype` is
+always required (e.g. "bug", "story", "task", etc). The set of required metadata
+is configurable (in JIRA) per `issuetype` so the set might be different between
+"bug" and "story", for example.
+
+For now, the bridge only supports exporting issues as a single `issuetype`. If
+no configuration is provied, then the default is `"id": "10001"` which is
+`"story"` in the default set of issue types.
+
+In addition to specifying the `issuetype` of issues created on export, the
+bridge will also allow you to specify a constant global set of default values
+for any additional required fields. See the configuration section below for the
+syntax.
+
+For longer term goals, see the section below on workflow validation
+
+### Assign git-bug id to field during issue creation
+
+JIRA allows for the inclusion of custom "fields" in all of their issues. The
+JIRA bridge will store the JIRA issue "id" for any bugs which are synchronized
+to JIRA, but it can also assign to a custom JIRA `field` the `git-bug` id. This
+way the `git-bug` id can be displayed in the JIRA web interface and certain
+integration activities become easier.
+
+See the configuration section below on how to specify the custom field where the
+JIRA bridge should write this information.
+
+
+### Workflows and Transitions
+
+JIRA issue states are subject to customizable "workflows" (project managers
+apparently validate themselves by introducing developer friction). In general,
+issues can only transition from one state to another if there is an edge between
+them in the state graph (a.k.a. "workflow"). JIRA calls these edges
+"transitions". Furthermore, each transition may include a set of mandatory
+fields which must be set in order for the transition to succeed. For example the
+transition of `"status"` from `"In Progress"` to `"Closed"` might required a
+`"resolution"` (i.e. `"Fixed"` or `"Working as intended"`).
+
+Dealing with complex workflows is going to be challenging. Some long-term
+aspirations are described in the section below on "Workflow Validation".
+Currently the JIRA bridge isn't very smart about transitions though, so you'll
+need to tell it what you want it to do when importing and exporting a state
+change (i.e. to "close" or "open" a bug). Currently the bridge accepts
+configuration options which map the two `git-bug` statuses ("open", "closed") to
+two JIRA statuses. On import, the JIRA status is mapped to a `git-bug` status
+(if a mapping exists) and the `git-bug` status is assigned. On export, the
+`git-bug` status is mapped to a JIRA status and if a mapping exists the bridge
+will query the list of available transitions for the issue. If a transition
+exists to the desired state the bridge will attempt to execute the transition.
+It does not currently support assigning any fields during the transition so if
+any fields are required the transition will fail during export and the status
+will be out of sync.
+
+### JIRA Changelog
+
+Some operations on JIRA issues are visible in a timeline view known as the
+`changelog`. The JIRA cloud product provides an
+`/issue/{issueIdOrKey}/changelog` endpoint which provides a paginated view but
+the JIRA server product does not. The changelog is visible by querying the issue
+with the `expand=changelog` query parameter. Unfortunately in this case the
+entire changelog is provided without paging.
+
+Each changelog entry is identified with a unique string `id`, but within a
+single changelog entry is a list of multilple fields that are modified. In other
+words a single "event" might atomically change multiple fields. As an example,
+when an issue is closed the `"status"` might change to `"closed"` and the
+`"resolution"` might change to `"fixed'`.
+
+When a changelog entry is imported by the JIRA bridge, each individual field
+that was changed is treated as a separate `git-bug` operation. In other words a
+single JIRA change event might create more than one `git-bug` operation.
+
+However, when a `git-bug` operation is exported to JIRA it will only create a
+single changelog entry. Furthermore, when we modify JIRA issues over the REST
+API JIRA does not provide any information to associate that modification event
+with the changelog. We must, therefore, herustically match changelog entries
+against operations that we performed in order to not import them as duplicate
+events. In order to assist in this matching proceess, the bridge will record the
+JIRA server time of the response to the `POST` (as reported by the `"Date"`
+response header). During import, we keep an iterator to the list of `git-bug`
+operations for the bug mapped to the Jira issue. As we walk the JIRA changelog,
+we keep the iterator pointing to the first operation with an annotation which is
+*not before* that changelog entry. If the changelog entry is the result of an
+exported `git-bug` operation, then this must be that operation. We then scan
+through the list of changeitems (changed fields) in the changelog entry, and if
+we can match a changed field to the candidate `git-bug` operation then we have
+identified the match.
+
+### Unlogged Changes
+
+Comments (creation and edition) do not show up in the JIRA changelog. However
+JIRA reports both a `created` and `updated` date for each comment. If we
+import a comment which has an `updated` and `created` field which do not match,
+then we treat that as a new comment edition. If we do not already have the
+comment imported, then we import an empty comment followed by a comment edition.
+
+Because comment editions are not uniquely identified in JIRA we identify them
+in `git-bug` by concatinating the JIRA issue `id` with the `updated` time of
+the edition.
+
+### Workflow Validation (future)
+
+The long-term plan for the JIRA bridge is to download and store the workflow
+specifiations from the JIRA server. This includes the required metadata for
+issue creation, and the status state graph, and the set of required metadata for
+status transition.
+
+When an existing `git-bug` is initially marked for export, the bridge will hook
+in and validate the bug state against the required metadata. Then it will prompt
+for any missing metadata using a set of UI components appropriate for the field
+schema as reported by JIRA. If the user cancels then the bug will not be marked
+for export.
+
+When a bug already marked for JIRA export (including those that were imported)
+is modified, the bridge will hook in and validate the modification against the
+workflow specifications. It will prompt for any missing metadata as in the
+creation process.
+
+During export, the bridge will validate any export operations and skip them if
+we know they will fail due to violation of the cached workflow specification
+(i.e. missing required fields for a transition). A list of bugs "blocked for
+export" will be available to query. A UI command will allow the user to inspect
+and resolve any bugs that are "blocked for export".
+
+## Configuration
+
+As mentioned in the notes above, there are a few optional configuration fields
+that can be set beyond those that are prompted for during the initial bridge
+configuration. You can set these options in your `.git/config` file:
+
+### Issue Creation Defaults
+
+The format for this config entry is a JSON object containing fields you wish to
+set during issue creation when exproting bugs. If you provide a value for this
+configuration option, it must include at least the `"issuetype"` field, or
+the bridge will not be able to export any new issues.
+
+Let's say that we want bugs exported to JIRA to have a default issue type of
+"Story" which is `issuetype` with id `10001`. Then we will add the following
+entry to our git-config:
+
+```
+create-issue-defaults = {"issuetype":"10001"}
+```
+
+If you needed an additional required field `customfield_1234` and you wanted to
+provide a default value of `"default"` then you would add the following to your
+config:
+
+```
+create-issue-defaults = {"issuetype":"10001","customfield_1234":"default"}
+```
+
+Note that the content of this value is merged verbatim to the JSON object that
+is `POST`ed to the JIRA rest API, so you can use arbitrary valid JSON.
+
+
+### Assign git-bug id to field
+
+If you want the bridge to fill a JIRA field with the `git-bug` id when exporting
+issues, then provide the name of the field:
+
+```
+create-issue-gitbug-id = "customfield_5678"
+```
+
+### Status Map
+
+You can specify the mapping between `git-bug` status and JIRA status id's using
+the following:
+```
+bug-id-map = {\"open\": \"1\", \"closed\": \"6\"}
+```
+
+The format of the map is `<git-bug-status-name>: <jira-status-id>`. In general
+your jira instance will have more statuses than `git-bug` will and you may map
+more than one jira-status to a git-bug status. You can do this with
+`bug-id-revmap`:
+```
+bug-id-revmap = {\"10109\": \"open\", \"10006\": \"open\", \"10814\": \"open\"}
+```
+
+The reverse map `bug-id-revmap` will automatically include the inverse of the
+forward map `bug-id-map`.
+
+Note that in JIRA each different `issuetype` can have a different set of
+statuses. The bridge doesn't currently support more than one mapping, however.
+Also, note that the format of the map is JSON and the git config file syntax
+requires doublequotes to be escaped (as in the examples above).
+
+### Full example
+
+Here is an example configuration with all optional fields set
+```
+[git-bug "bridge.default"]
+	project = PROJ
+	credentials-file = .git/jira-credentials.json
+	target = jira
+	server = https://jira.example.com
+	create-issue-defaults = {"issuetype":"10001","customfield_1234":"default"}
+	create-issue-gitbug-id = "customfield_5678"
+	bug-open-id = 1
+	bug-closed-id = 6
+```
+
+## To-Do list
+
+* [0cf5c71] Assign git-bug to jira field on import
+* [8acce9c] Download and cache workflow representation
+* [95e3d45] Implement workflow gui
+* [c70e22a] Implement additional query filters for import
+* [9ecefaa] Create JIRA mock and add REST unit tests
+* [67bf520] Create import/export integration tests
+* [1121826] Add unit tests for utilites
+* [0597088] Use OS keyring for credentials
+* [d3e8f79] Don't count on the `Total` value in paginations
+
+
+## Using CURL to poke at your JIRA's REST API
+
+If you need to lookup the `id` for any `status`es or the `schema` for any
+creation metadata, you can use CURL to query the API from the command line.
+Here are a couple of examples to get you started.
+
+### Getting a session token
+
+```
+curl \
+  --data '{"username":"<username>", "password":"<password>"}' \
+  --header "Content-Type: application/json" \
+  --request POST \
+  <serverUrl>/rest/auth/1/session
+```
+
+**Note**: If you have a json pretty printer installed (`sudo apt install jq`),
+pipe the output through through that to make things more readable:
+
+```
+curl --silent \
+  --data '{"username":"<username>", "password":"<password>"}' \
+  --header "Content-Type: application/json" \
+  --request POST
+  <serverUrl>/rest/auth/1/session | jq .
+```
+
+example output:
+```
+{
+  "session": {
+    "name": "JSESSIONID",
+    "value": "{sessionToken}"
+  },
+  "loginInfo": {
+    "loginCount": 268,
+    "previousLoginTime": "2019-11-12T08:03:35.300-0800"
+  }
+}
+```
+
+Make note of the output value. On subsequent invocations of `curl`, append the
+following command-line option:
+
+```
+--cookie "JSESSIONID={sessionToken}"
+```
+
+Where `{sessionToken}` is the output from the `POST` above.
+
+### Get a list of issuetype ids
+
+```
+curl --silent \
+  --cookie "JSESSIONID={sessionToken}" \
+  --header "Content-Type: application/json" \
+  --request GET https://jira.example.com/rest/api/2/issuetype \
+   | jq .
+```
+
+**example output**:
+```
+  {
+    "self": "https://jira.example.com/rest/api/2/issuetype/13105",
+    "id": "13105",
+    "description": "",
+    "iconUrl": "https://jira.example.com/secure/viewavatar?size=xsmall&avatarId=10316&avatarType=issuetype",
+    "name": "Test Plan Links",
+    "subtask": true,
+    "avatarId": 10316
+  },
+  {
+    "self": "https://jira.example.com/rest/api/2/issuetype/13106",
+    "id": "13106",
+    "description": "",
+    "iconUrl": "https://jira.example.com/secure/viewavatar?size=xsmall&avatarId=10316&avatarType=issuetype",
+    "name": "Enable Initiatives on the project",
+    "subtask": true,
+    "avatarId": 10316
+  },
+  ...
+```
+
+
+### Get a list of statuses
+
+
+```
+curl --silent \
+  --cookie "JSESSIONID={sessionToken}" \
+  --header "Content-Type: application/json" \
+  --request GET https://jira.example.com/rest/api/2/project/{projectIdOrKey}/statuses \
+   | jq .
+```
+
+**example output:**
+```
+[
+  {
+    "self": "https://example.com/rest/api/2/issuetype/3",
+    "id": "3",
+    "name": "Task",
+    "subtask": false,
+    "statuses": [
+      {
+        "self": "https://example.com/rest/api/2/status/1",
+        "description": "The issue is open and ready for the assignee to start work on it.",
+        "iconUrl": "https://example.com/images/icons/statuses/open.png",
+        "name": "Open",
+        "id": "1",
+        "statusCategory": {
+          "self": "https://example.com/rest/api/2/statuscategory/2",
+          "id": 2,
+          "key": "new",
+          "colorName": "blue-gray",
+          "name": "To Do"
+        }
+      },
+...
+```