[Architecture] Data loaders for loading data into CouchDB

Tony Atkins tony at raisingthefloor.org
Wed Aug 10 07:20:13 UTC 2016


Hi, All:

Happy to work this into our ongoing conversations around PouchDB and the
gpii-pouchdb package.   Right now we use methods unique to PouchDB to load
data, but we could certainly use a similar or even the same data loader
there, at least for the grades that use express-pouchdb.  The browser side
would need a little more work, but hopefully with a general enough concept
for the data loader, we would still end up with one central concept, well
exercised in a few packages.

Cheers,


Tony

On Tue, Aug 9, 2016 at 3:39 PM, Li, Cindy <cli at ocadu.ca> wrote:

> Hi,
>
> I’m working on using CouchDB for the auth server data persistence when
> GPII is running in the production mode. Avtar, Simon and I were going thru
> the deployment of the preferences server VM that is already backed up using
> CouchDB when running in the production mode.
>
> One thing we noticed is, currently the data loading process for importing
> the initial perfs data into the CouchDB, including the preprocessing of
> converting the prefs data into CouchDB compatible data structure, is
> performed by the docker image - https://github.com/gpii-ops/
> docker-preferences-server-data-loader
>
> More particularly, 2 ansible roles that are called up by the docker:
>
> https://github.com/gpii-ops/ansible-preferences-server-data-loader
> https://github.com/gpii-ops/ansible-preferences-server
>
> This means scripts written in ansible roles need to be aware of where the
> data, as well as the GPII production config file, are located in the GPII
> universal repo, its data structure, how to convert that data structure into
> CouchDB sensible structure etc. All these are currently hardcoded in
> ansible scripts so that any change on these information could break the
> data loading process. One existing breakage is https://issues.gpii.net/
> browse/GPII-1884, which is caused by the renaming of the GPII production
> config file.
>
> To solve this issue, Avtar, Simon and I would like to propose creating
> data loaders in the GPII universal repo. These data loaders can be
> called/executed by external resources, the docker image in our case, to
> load data into CouchDB. This would help to isolate the universal specific
> info within the universal repo, reduce dependencies btw the docker VM and
> the GPII universal, also reduce the amount of work required on the docker
> image.
>
> At the moment, 2 data loaders are needed:
>
> 1. To convert and load prefs data into CouchDB for the preferences server;
> 2. To load data and views into CouchDB for the auth server.
>
> Your ideas and suggestions are appreciated. We can also discuss this in
> tomorrow’s arch meeting.
>
> Cindy
>
>
> _______________________________________________
> Architecture mailing list
> Architecture at lists.gpii.net
> http://lists.gpii.net/mailman/listinfo/architecture
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gpii.net/pipermail/architecture/attachments/20160810/efc7e411/attachment-0001.html>


More information about the Architecture mailing list