[Architecture] The State of our Branches

Antranig Basman antranig.basman at colorado.edu
Wed Nov 5 08:42:19 EST 2014


Kaspar and I have been at work for some months on significant 
improvements to our core architecture - unfortunately during this time 
it has not been possible to process new contributions. This set of 
long-term branches are now getting very close to being merged so I 
thought I would provide some notes regarding the changes and also some 
details about the merging endgame, timeline and last requirements.

My work has mostly been under the headline of 
http://issues.gpii.net/browse/GPII-434 - which in turn was revealed as a 
requirement some months ago in order to be able to merge Simon Bates' 
work for GPII-94 enabling us to upgrade to current versions of node.js 
on Windows. As Steve Lee will attest, our current set of version 
requirements on Windows are sufficiently behind the curve as to make 
installing the GPII on Windows quite intricate and error-prone. This 
work also enables the use of "Integration Tests" - the configuration 
previously applied as "Acceptance Tests" in the architecture-specific 
branches has been hoisted up into the universal project, allowing us to 
exercise the complete workflow of the GPII system for these scenarios in 
a portable way, allowing us to resolve any issues that they raise 
separately from the action of the settings handlers and solution 
launchers on the real systems.

Kaspar's work is under the headline of 
http://issues.gpii.net/browse/GPII-941 - this makes improvements to our 
MatchMaker payloads that will make MatchMaker integration significantly 
easier in future (previous integrations have taken wholesale forks of 
universal) as well as preparing for the work required for the 
Context-Aware Server and Environmental Reporter. This work also makes 
essential improvements to the Ontology Server which will be necessary, 
for example, to enable security profile settings in the PMT as well as 
simplifying MatchMaker implementation.

My branches for this work are at
https://github.com/amb26/universal/tree/GPII-434
with a corresponding Windows branch at
https://github.com/amb26/windows/tree/GPII-434

These are now at the stage where they function "roughly as well" as the 
old system but unfortunately the improved settings handler and lifecycle 
manager have shed some new light on the exact causes of our test 
failures on Windows which prompted this work in the first place.

Kaspar's branch is at
https://github.com/GPII/universal/pull/279/files which has been merged 
up with my branch as of about a week ago.

The new workflow enables the deployment of settings handlers which are 
i) asynchronous, and ii) which the system will automatically retry until 
the required result is received. Comparing the results of successive 
settings snapshots on Windows has revealed a very peculiar effect, which 
I can only explain in terms of a cross-linkage between the effects of 
different settings handlers on Windows.

PiratePad seems down at the moment but a sample transcript can hopefully 
seen in future at
http://piratepad.net/failed-transcript
This shows the Windows cursor settings apparently snapshotted in the 
middle of what must be an OS-initiated process to reset their values. 
Three of them hold their original values and the other 10 have been 
reset. On the following snapshot, all 13 have been reset. My belief is 
that this process is initiated asynchronously by the OS in response to a 
desktop theme change - which itself is a change of which we have no 
means currently of detecting the time point of application (see GPII-581).
So a little further work is required in order to produce acceptance 
tests which pass "essentially all the time" - I will need to improve 
this branch so that read retries also involve write retries. However I 
think this implementation is close enough to the final one that we 
should start coordinating the final phases of the merging process.

Some important changes in the GPII-434 branches:

i) As revealed by GPII-1001, many of our settings handlers had faulty 
implementations for the "get" cycle, as well as in general having a 
faulty or absent idiom for the "removal" of values that had previously 
been found to be unset when the system started up. Although it is 
architecturally dubious, the quickest way of resolving this issue 
without completely disrupting our settings handler contract has been to 
permit the appearance of the JavaScript "undefined" value as a settings 
handler payload value. When re-presented with this value, the "set" 
cycle will interpret this as an instruction to entirely remove the 
corresponding key.
This means that our settings handler payloads have become invalid when 
considered as JSON values. The sounder solution in the long term is to 
move our settings handler API over towards the form operated in 
Infusion's ChangeApplier API -
https://github.com/fluid-project/infusion-docs/blob/master/src/documents/ChangeApplierAPI.md#programmatic-style-1
as well as allowing us to account for value deletion in a stable way, it 
will also allow in time for encoding other important operations such as 
insertion of elements into an array. In the sort term, if we find the 
requirement for transmitting or storing these payloads, they will need 
to go through a separate phase of escaping and de-escaping in order to 
account for their status as improper JSON. We may imminently find 
ourselves doing this in upcoming work on what could be called the 
"WebSockets (inverse) Settings Handler" for communicating with GPII 
browser plugins/extensions in the role of a settings handler.

Javi - in the meantime, for GPII-1001, you can adopt the workflow in the 
gpii.setttingsHandlers.setSettings method at
https://github.com/amb26/universal/blob/GPII-434/gpii/node_modules/settingsHandlers/src/settingsHandlerUtilities.js#L229
  - this uses an Infusion shorthand for accounting for the difference 
between settings writing and deletion

ii) Wholesale removal of the "LifecycleManagerServer" which has been 
found to be a redundant architectural component

iii) As mentioned before, the LifecycleManager has a fully asynchronous 
workflow based on "micropromises" (introduced in FLUID-5513), and all 
test cases previously taking the form of "acceptance tests" are now 
available as "integration tests" (runnable via IntegrationTests.js) at
https://github.com/amb26/universal/blob/GPII-434/tests/IntegrationTests.js

This branch still contains a moderate amount of debugging code but I 
think it is stable enough for Kaspar to merge up his branch with again. 
In particular there will be changes to the LifecycleManager which will 
cause conflicts - I've moved this over to an invoker-based model since 
it would be too awkward to inject in the "retrying configuration" 
otherwise. Also the removal of the "LifecycleManagerServer" will cause 
conflicts with your current work on fixing the "update" action for the 
new MatchMaker semantic.

iv) Others (e.g. the new standalone drivers named "gpii.js", the new 
module resolution system that eliminates hard-coded file references 
between modules, improvements to grunt scripts and linting for all 
projects, the "nameResolver" system for locating mocks for settings 
handlers and actions, etc.)

Colin is at work reviewing the underlying pull requests to Infusion and 
Kettle that all this work depends on - given the large-scale nature of 
the changes to GPII I think it makes sense for us to adopt a process of 
"collective review" for the core architecture changes since the branches 
contain work from so many different contributors.

I and Javi still need to produce a corresponding branch of the Linux 
repository which syncs up with this work - this depends on having 
GPII-1001 resolved first - the other changes in packaging (e.g. removal 
of acceptance tests) should be relatively straightforward. There will 
also need to be a branch of the android repo too.

More about this at arch meeting later -

Cheers,
Antranig


More information about the Architecture mailing list