Upgrade Instructions 6.0.0 - Migrating MAM Tags from Mongo to ArangoDB 

Ooyala Flex has 2 type of tags:

  • User defined tags (Tags are added as a primitive in Metadata designer)
  • Account tags (Old MAM tags will be called as Account tags)

In order to migrate existing MAM tags from MongoDB to ArangoDB, the Tags migrator has been written. Follow the instructions specified below to start the migration.

Download the Migrator JAR File

You can find the migrator in artifactory: https://artifactory.nativ-systems.com/artifactory/webapp/#/artifacts/browse//search/quick/eyJzZWFyY2giOiJxdWljayIsInF1ZXJ5IjoiKnRhZ3MtbWlncmF0aW9uKmphciIsInNlbGVjdGVkUmVwb3NpdG9yaWVzIjpbXX0=

Ensure that you download the latest non-snapshot version (eg. tags-migration-0.0.1-jar-with-dependencies.jar)

Create configuration file

Download database-migration.properties and edit to make sure all the properties are set correctly:

The mongo.host property needs to point at the current Mongo primary. You can find the IP address of the primary by running the following command:

mongo --host your-mongo-host --eval 'rs.status().members.find(r=>r.state===1).name'

The ArangoDB port needs to be 7001 (it has to point at a coordinator ArangoDB node in a distributed ArangoDB installation. The host can be any host running an instance of ArangoDB coordinator.

See the migrator configuration section below for a full description of configuration options.

Launch the application

            java -jar tags-migration-0.0.1-jar-with-dependencies.jar -c database-migration.properties
        

You should see a message similar to this:

2017-08-14 13:33:23.073 [main] INFO  c.flex.tags.migration.MongoToArango - Config : Config: {arango.db.no.auth.enabled=false, arango.port=7001, arango.collection.objects=objects, mongo.port=27017, mongo.projection={'objectId':1, 'objectType':1, 'name':1, 'accountId':1, 'workspaceId':1, 'userId': 1}, arango.db=flex-tag, mongo.host=localhost, arango.host=localhost, mongo.batch_size=20, arango.collection.tagCollections=tagCollections, mongo.query={}, arango.collection.tags=tags, arango.user.password=password, mongo.db=flex-tag, arango.graph.edge=objectTagsEdge, arango.user.name=root, mongo.collection=tags, arango.graph=objectTagsRelationships}
2017-08-14 13:33:23.185 [main] INFO  org.mongodb.driver.cluster - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
2017-08-14 13:33:23.268 [main] INFO  org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2017-08-14 13:33:23.290 [cluster-ClusterId{value='5991a6a33949bf5f19b0b28e', description='null'}-localhost:27017] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:19891}] to localhost:27017
2017-08-14 13:33:23.292 [cluster-ClusterId{value='5991a6a33949bf5f19b0b28e', description='null'}-localhost:27017] INFO  org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 4]}, minWireVersion=0, maxWireVersion=5, electionId=7fffffff000000000000007a, maxDocumentSize=16777216, roundTripTimeNanos=953675, setName='qa3', canonicalAddress=10.20.15.35:27017, hosts=[10.20.15.35:27017, 10.20.15.73:27017, 10.20.15.164:27017], passives=[], arbiters=[], primary='10.20.15.35:27017', tagSet=TagSet{[]}}
2017-08-14 13:33:23.491 [main] INFO  c.flex.tags.migration.MongoToArango - Skipping database 'flex-tag' creation as it already exists
2017-08-14 13:33:23.504 [main] INFO  c.flex.tags.migration.MongoToArango - Collection 'tagCollections' already exists
2017-08-14 13:33:23.510 [main] INFO  c.flex.tags.migration.MongoToArango - Collection 'tags' already exists
2017-08-14 13:33:23.517 [main] INFO  c.flex.tags.migration.MongoToArango - Collection 'objects' already exists
2017-08-14 13:33:23.534 [main] INFO  c.flex.tags.migration.MongoToArango - Start : Mon Aug 14 13:33:23 UTC 2017
2017-08-14 13:33:23.664 [main] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:19892}] to localhost:27017
       

At this point the messages will stop. Once the migration is complete you should see a message like this:

2017-08-14 13:33:24.078 [main] INFO  c.flex.tags.migration.ArangoMigrator - Total of rejected tags : 0
2017-08-14 13:33:24.078 [main] INFO  c.flex.tags.migration.ArangoMigrator - Total of tag documents migrated : 8
2017-08-14 13:33:24.078 [main] INFO  c.flex.tags.migration.ArangoMigrator - Total of inserted docs in `tagCollections` collection: 2
2017-08-14 13:33:24.078 [main] INFO  c.flex.tags.migration.ArangoMigrator - Total of account tags collections already exists : 2
2017-08-14 13:33:24.078 [main] INFO  c.flex.tags.migration.ArangoMigrator - Total of inserted docs in `tags` collection : 7
2017-08-14 13:33:24.078 [main] INFO  c.flex.tags.migration.ArangoMigrator - Total of inserted docs in `objects` collection : 8
2017-08-14 13:33:24.078 [main] INFO  c.flex.tags.migration.MongoToArango - End : Mon Aug 14 13:33:24 UTC 2017
   

This means the migration is now complete.

Migration properties and configurations

Property Default Value Description
arango.collection.objects (none) Flex objects collection name in ArangoDB. The objects collection name is 'objects'
arango.collection.tagCollections (none) The Tag collection's collection name in ArangoDB. The tag collections collection name in Arango is 'tagCollections'
arango.collection.tags (none) The Tags collection name in ArangoDB. The tag collections collection name in Arango is 'tags'
arango.db test ArangoDB name
arango.graph (none) The relationship graph that is used to store object and tag associations. The graph name is 'objectTagsRelationships'
arango.graph.edge (none) The edge that is used to generate graphs. The edge collection name is 'objectTagsEdge'
arango.host localhost The host address of the ArangoDB
arango.port 8529 The port of the ArangoDB
mongo.batch_size 20 Batch size for the find query
mongo.collection test Collection name to be migrated (In our case the collection name will be tags)
mongo.db test Tag Mongo Database Name. In our case the database name is 'flex-tag'
mongo.host localhost The host address of the MongoDB
mongo.port 27017 The port of the MongoDB
mongo.projection (none) In our case the projection value is{'objectId':1, 'objectType':1, 'name':1, 'accountId':1, 'workspaceId':1, 'userId': 1}
mongo.query {} Query to select data

Migration Conditions:

1) Existing tags are MAM tags. While saving tags create tag collection based on the accountId provided in the tags. If account collection already exists ignore it just pick the tagCollectionId of the existing account tag collection and save it to tags.

2) The MAM tag migration library is a java program that will make a connection to mongo and query the tags document collection from mongo and it should save the tags to ArangoDB.

The old MAM tag model that we have in MongoDB is as follows:

 {
  "objectId": 111,
  "objectType": "media-asset",
  "name": "ooyala flex",
  "accountId": 1,
  "workspaceId": 123,
  "userId": 222
}  
        

While migrating this Arango it should create a ACCOUNT tag collection in tagCollections collection and save the tag in tags collection. After successful migration the created ACCOUNT Tag Collection in tagCollections collection:

{
  "id": 123456789,           // ID of the tag collection
  "name": "account-tag-collection-1",
  "displayName": "account tag collection 1",
  "type": "ACCOUNT",
  "enabled": true,
  "accountId": 1,
  "created": "26 Jun 2017 07:34:02 +00:00",
  "lastModified": "26 Jun 2017 07:34:02 +00:00"
}     
       

A tag will be created in tags collection:

{
  "tagCollectionId": 123456789,  // created ACCOUNT tag collection ID
  "tag": "ooyala flex"
}  
        

The Object will be stored in objects collection as:

{
  "objectId": 11,
  "objectType": "media-asset",
  "objectTypeId": 2
}        
        
Note: objectTypeId will be taken from objectType

While storing the object an edge will be created from the created tag Id to created object Id to associate both.

Note: If Account tag collection already exists it should not create a new Account tag collection in database.
Note: If tag already exists for a given accountId, duplicates will not be created
Note: If Objects already exists, duplicates will not be created
Note: MAM tags allowed any kind of tags. But the migrator will not migrate tags if it doesn't follow new the standards.
https://help.ooyala.com/sites/all/libraries/dita/en/media-logistics/flex/dev/60/upgrade_instructions_60_migrating_mam_tags_from_mongo.html

Was this article helpful?