Adding to this with a user's perspective (per Katarina's suggestion):

I've found so far that in multiple scenarios where one is performing write operations and there is a one-to-many relationship built into one's primary data model, one needs to rely on computationally less efficient workarounds due to the lack of current support for the FOREACH clause. A key example here is of course Twitter. There are several different popular tools that source data from Twitter (whether it's Twint, the official firehose API, or dataset CSVs directly released by the Twitter Trust & Safety department in the case of state-sponsored information operations that their internal staff have found), and although those tools have some differences in their data schema, they all generally output data in the form of tabular rows of tweets. Across all the tools I've seen, the tweet schema always contains columns that define a one-to-many relationship for the given property of a Tweet, such as hashtags, urls, and user_mentions (or their equivalents). For example, below is a tweet from a troll account that was operated by the Russian Internet Research Agency and which was released by Twitter in their Northern Africa IO dataset:

tweetid,userid,user_display_name,user_screen_name,user_reported_location,user_profile_description,user_profile_url,follower_count,following_count,account_creation_date,account_language,tweet_language,tweet_text,tweet_time,tweet_client_name,in_reply_to_userid,in_reply_to_tweetid,quoted_tweet_tweetid,is_retweet,retweet_userid,retweet_tweetid,latitude,longitude,quote_count,reply_count,like_count,retweet_count,hashtags,urls,user_mentions
3.76047909948883E+017,FDAiQiDNukYhjuXtuoj8oXCk9DaeuDk6dWKCID+Kscc=,FDAiQiDNukYhjuXtuoj8oXCk9DaeuDk6dWKCID+Kscc=,FDAiQiDNukYhjuXtuoj8oXCk9DaeuDk6dWKCID+Kscc=,,,,51,398,2013-07-01,en,en,RT @UNPeacekeeping: AU-#UN Mission in #Darfur recovers body of one of four missing peacekeepers swept away by floods http://t.co/OdB4aNllE6…,2013-09-06 18:23,Mobile Web (M2),,,,True,,376029614071874,absent,absent,0,0,0,0,"['UN', 'Darfur']",['http://ow.ly/oDnGq'],"['46003622']"

To load a CSV file of bulk Twitter data with the above schema into Neo4J, one might use the following Cypher queries (credit to Adam Crowley and his repo at https://github.com/adam-cowley/twittergraph):

MERGE (t:Tweet {id_str:status.id_str})
SET t.text=status.text,t.created_at=status.created_at,t.retweet_count=status.retweet_count, t.favorite_count=status.favorite_count

// Create Author
MERGE (u:User {screen_name:user.screen_name})
SET u.name = user.name, u.friends_count = user.friends_count, u.followers_count = user.followers_count, u.picture=user.profile_image_url

MERGE (u)-[:TWEETED]->(t)

// Create Hashtags
FOREACH (h in entities.hashtags | MERGE (ht:Hashtag {name:h.text}) MERGE (t)-[:MENTIONS_HASHTAG]->(ht))

// Mentions
FOREACH (m in entities.user_mentions | MERGE (mu:User {screen_name:m.screen_name})  MERGE (t)-[:MENTIONS_USER]->(mu))

// URLs
FOREACH (m in entities.urls | MERGE (mu:URL {url:m.url})  MERGE (t)-[:MENTIONS_URL]->(mu))
.....

and so on.

To accomplish the same task in Memgraph, I am able to use the same logic for creating the core Tweet, the user who authored the tweet, and other nodes that have a 1:1 relationship with the Tweet. However, as we have discussed on Discord in the past, I must use triggers to create nodes from the list properties of each Tweet due to the lack of FOREACH support - one trigger for each list property. This is a more involved and laborious workaround than it looks at first glance, because when each trigger executes in the database, the JSON record object for the given tweet (the one that was received from Kafka and used to generate the original Cypher write statement above) is no longer in scope and cannot be accessed. As a consequence of this, I must write the list of values for the given property onto the Tweet node during initial creation, so that the values list is present on each element in the returned createdVertices collection and thus is passed correctly to the given trigger:

                mgp.Record(
                    query=(f'''MERGE (t:Tweet{
                            ":Retweet" if tweet_dict["is_retweet"] else ""
                        } {{id: $tweet_id}}) 
                        SET t.created_at=localdatetime($created_at), 
                        t.text=$text, t.replies_count=$replies_count, 
                        t.retweets_count=$retweets_count, t.quote_count=$quote_count, t.like_count=$like_count,
                        t.language=$tweet_language, **_t._hashtags=$hashtags_**,
                        **_t._mentions=$user_mentions_**{", **_t._urls=$tweet_urls_**" if tweet_dict["is_retweet"] is False and tweet_dict["urls"] else ""}
                        

                        '''),

**_ emphasis mine on the relevant Tweet properties, even though the code formatting doesn't let me bold & italicize them like I'd hoped. _**
These list properties aren't really supposed to be part of the graph data model for a Tweet, though - they're supposed to be their own node category. That's why I have prefixed each property with _ above, to denote them as something like a private property in a class. To some extent, this seems like an anti-pattern that goes against the intended usage of graph databases like Memgraph.

After each created Tweet is committed to the database, the three triggers then call the corresponding custom write procedures, which is an extra computational step that isn't always needed (like if the tweet has no hashtags or URLs and mentioned no users):
memgraph.execute( '''CREATE TRIGGER created_tweet_trigger_mentions ON () CREATE AFTER COMMIT EXECUTE CALL twitter_procs.create_mentions(createdVertices)''' )

Then we get to the custom write_procedure implementations themselves. It is true that iterating over things within such modules is easy enough, but you also have to query and iterate over the whole existing graph for the set of "child" nodes that you want to MERGE into. One can do that first in a Cypher clause within the trigger definition, e.g. ON () CREATE AFTER COMMIT EXECUTE MATCH (n :Hashtag) WITH COLLECT (n.name, n.id) AS hashtags_lookup_map CALL twitter_procs.create_hashtags(createdVertices, hashtags_lookup_map), or in Python etc within the code of the custom write procedure, but you still have to do it as yet another step.

Here is example code from one of my workaround custom write-procedures:

@mgp.write_proc
def create_mentions(context: mgp.ProcCtx,
                    created_vertices: mgp.Any) -> mgp.Record():
    for vert in created_vertices:
        if vert.labels[0] == "Tweet" and "_mentions" in vert.properties.keys():
            user_mentions = vert.properties["_mentions"]    
            
            assert isinstance(user_mentions, tuple)
            if len(user_mentions) > 0:
                user_vertices_list = [vertex for vertex in context.graph.vertices if any(label == "User" for label in vertex.labels)]

                for i in range(0, len(user_mentions)):
                    # Note to self: name of the below is walrus operator or assignment expression
                    #target_user = _[0] if (_:= [x for x in user_vertices_list if x.properties["id"] == user_mentions[i]]) else None
                    _ = [x for x in user_vertices_list if x.properties["id"] == user_mentions[i]]
                    target_user = _[0] if len(_) > 0 else None
                    if target_user is not None:
                        edge = context.graph.create_edge(vert, target_user, mgp.EdgeType("MENTIONS"))
                    else:
                        target_user_vertex = context.graph.create_vertex()
                        target_user_vertex.add_label("User")
                        log_to_stderr_by_force(f"ID of new User vertex is: {target_user_vertex.id}")
                        target_user_vertex.properties["id"] = user_mentions[i]
                        edge = context.graph.create_edge(vert, target_user_vertex, mgp.EdgeType("MENTIONS"))
                        
    return mgp.Record()

The above must be repeated for each list property. And it gets called after every vertex creation event, whether or not the vertex is actually a Tweet.

It's rather a lot of code to have to write (and debug the first time around) when it could easily be accomplished by a one-line Cypher statement like FOREACH (m in entities.user_mentions | MERGE (mu:User {screen_name:m.screen_name}) MERGE (t)-[:MENTIONS_USER]->(mu)) if such syntax was supported.

P.S. Oh, by the way, while we're looking at that code - Memgraph's Python version doesn't seem to recognize the walrus operator that was introduced in Python 3.7 or 3.8, whichever version it was. That's the reason for the commented-out line of code.

The FOREACH language construct for performing an operation on every list element #308

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions