This is a repository of different scripts used at University of Lorraine to populate the collections of the OpenLRW API.

All these scripts are made for the Educational community, don't hesitate to contribute!

• Python Scripts

  • Python ≥ 2.7
  • openlrw-python-api-client
  • python-ldap
  • PyYAML
  • MySQLdb
  • OpenLDAP

• Logstash Scripts

  • Logstash ≥ 2.4

• User
• Event
• Class
• Result
• LineItem
• Enrollment

$ git clone https://github.com/xchopin/openlrw-scripts

$ cd OpenLRW-scripts/bootstrap
$ cp settings.yml.dist settings.yml ; vi settings.yml

To get the libraries you will need to have PIP package manager

• A. Download and install PIP

  $ wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py ; python /tmp/get-pip.py
  

• B. python-ldap

  ⚠️ This library requires python-devel and openldap

  $ pip install python-ldap
  

• C. PyYAML

  $ pip install pyyaml
  

• D. MySQLdb

  ⚠️ This library requires mariadb-devel yum install mariadb-devel

  $ pip install mysqlclient
  

• E. OpenLRW

  $ pip install openlrw
  

-v / --verbose : to print all the HTTP calls
-no-mail : to don't send any emails
-h / --help : help documentation

• Import all the users (populate)

This script will import the users by using the LDAP database.

Clear then populate the collection (recommended for a new OpenLRW instance)

$ python collections/Users/LDAP/import_users.py --reset

• Update the collection

Add the new users to the collection

$ python collections/Users/LDAP/import_users.py --update

The new information will be added into the metadata attribute of users.

• Parse a file

$ python collections/Users/Apogee/import_civic_information.py --file path_to_your_file.csv

• Parse the files containing the date of yesterday

Aimed to a CRON use - Format: YYYYmmdd.csv - eg: Students_CS_20191030.csv; Students_Biology_20191030.csv

⚠ The repository of your source files has to be indicated in the settings file, at the civic_information_directory attribute.

$ python collections/Users/Apogee/import_civic_information.py --file path_to_your_file.csv

CMP;LOGIN;YEAR;GENDER;CHILDREN;HANDICAP;HAS_SCHOLARSHIP;HAS_A_JOB;HAS_ADAPTED_STUDY_PLAN;CITY;BACCALAUREAT_YEAR;BACCALAUREAT_TYPE;BACCALAUREAT_ZIPCODE;BACCALAUREAT_HONOR
AB0;foo;1998;F;0;null;N;O;N;PARIS;2018;ES;054;P
AB0;bar;2000;F;0;null;N;O;N;NANCY;2017;ES;054;AB
AB0;foobar;1991;M;0;null;N;N;O;LYON;2009;S;093;P

This script will import the "logged-in" events (students only) by using log files

• For one log file

$ cd collections/Events/CAS/
$ cat /logs/cas_auth.log | /opt/logstash/bin/logstash --quiet -w10 -f authentication.conf

• Treating a plenty of log files (from a date to YESTERDAY)

$ cd collections/Events/CAS/
$ sh authentications.sh

• Import all the events

From a timestamp

$ python collections/Events/Moodle/import_events.py -t TIMESTAMP

From a timestamp to another one

$ python collections/Events/Moodle/import_events.py -t TIMESTAMP TIMESTAMP

• Update the events

Check the most recent event stored in the database and insert the newer ones

 $ python collections/Events/Moodle/import_events.py -u

This script imports the Moodle classes, it uses a .txt file to allow you to active the class you want (other will be inactive).
It also checks duplicates so you can run it several times for updates.

⚠ The template file is located at data/Classes/active_classes.txt.dist.

$ cp data/Classes/active_classes.txt.dist data/Classes/active_classes.txt

Then add the class id (one per line), you can add comments with the # character.
In order to set all your classes as active just let the file empty.

$ python collections/Classes/Moodle/import_classes.py

This script will clear the whole Class collection from MongoDB (it keeps the indices though), it performs only one HTTP DELETE request.

$ python collections/Classes/delete_all_classes.py

This script imports the Quizzes, Active quizzes and the Grades from the Moodle database. Checking method is used for HTTP Post so you can use the script for populating and updating your collection.

The condition is made on Moodle's timemodified attribute

$ python collections/Results/Moodle/import_results.py --from TIMESTAMP

The condition is made on Moodle's timemodified attribute

$ python collections/Results/Moodle/import_results.py --from TIMESTAMP --to TIMESTAMP

Import only the results that are older to the last result in MongoDB

$ python collections/Results/Moodle/import_results.py -u

Import the results from an Apogée csv file; it also add LineItem objects if they don't exist. In order to link these new LineItems to a Moodle class you will have a to use a script (more details in LineItems section)

⚠ Two field are required in the settings file

• results_directory where you have to put files with the following format YYYYmmdd.csv eg: export1_20193010.csv
• lineitems_name_filepath you have to indicate the absolute path of a csv file for the lineitems

foobar1;2019;3WLAEIMIA;200;initiale;1WIN12;800;UE1-AAAAA1-17.0-ADM
foobar2;2019;3WLAEIMIA;200;initiale;1WIN12;800;UE1-AAAAA1-12.0-ADM
foobar2;2019;3WLOEIMIA;200;initiale;1WIN12;800;UE1-AAAAA3-12.0-null

COD_ELP;LIB_ELP
AAAAA1;Discret Math
AAAAA2;Web applications with JavaEE
AAAAA3;Management
AAAAA4;Internship

Import all the results from the files containing the date of yesterday

$ python collections/Results/Apogee/import_results.py -l

Make a difference between the results of yesterday and the day before (for CRON)

$ python collections/Results/Apogee/import_results.py -u

This script imports the lineItems from Moodle and also checks duplicates, it means you can use it several times for updates.

Import the LineItems

$ python collections/LineItems/Moodle/import_lineitems.py

This script maps the LineItem objects created by the import_results.py script. For each Moodle class that contains the classCode key in metadata it checks if it equals their sourcedId. Ita lso checks duplicates which means you can use it several times for updates.

Map the Apogée LineItems to Moodle classes

$ python collections/LineItems/Apogee/map_classes.py

This script imports the enrollments from Moodle with a timestamp argument (from)

Import the Enrollments

$ python collections/Enrollments/Moodle/import_enrollments.py --from TIMESTAMP

Update the Enrollments

$ python collections/Enrollments/Moodle/import_enrollments.py -u

This script will clear the whole Enrollment collection from MongoDB (it keeps the indices though), it performs only one HTTP DELETE request.

$ python collections/Enrollments/delete_all_enrollments.py

OpenLRW-scripts is made available under the terms of the Educational Community License, Version 2.0 (ECL-2.0).