Introduction
In Intellos, you can use some crawler tools, such as Jina, which can convert web pages into markdown format that LLMs can read. Recently, wordware.ai has brought to our attention that we can use crawlers to scrape social media for LLM analysis, creating more interesting applications. However, knowing that X (formerly Twitter) stopped providing free API access on February 2, 2023, and has since upgraded its anti-crawling measures. Tools like Jina are unable to access X’s content directly.Starting February 9, we will no longer support free access to the Twitter API, both v2 and v1.1. A paid basic tier will be available instead 🧵 — Developers (@XDevelopers) February 2, 2023Fortunately, Intellos also has an HTTP tool, which allows us to call external crawling tools by sending HTTP requests. Let’s get started!
Prerequisites
Register Crawlbase
Crawlbase is an all-in-one data crawling and scraping platform designed for businesses and developers. Moreover, using Crawlbase Scraper allows you to scrape data from social platforms like X, Facebook and Instagram. Click to register: crawlbase.comIntellos workspace
You need access to an Intellos workspace, for example https://app.intellos.ru.Configure LLM Providers
Configure Model Provider in account setting:
Create a chatflow
Now, let’s get started on the chatflow. Click onCreate from Blank to start:
Add nodes to chatflow
Start node
In start node, we can add some system variables at the beginning of a chat. In this article, we need a Twitter user’s ID as a string variable. Let’s name itid .
Click on Start node and add a new variable:
Code node
According to Crawlbase docs, the variableurl (this will be used in the following node) should be https://twitter.com/ + user id , such as https://twitter.com/elonmusk for Elon Musk.
To convert the user ID into a complete URL, we can use the following Python code to integrate the prefix https://twitter.com/ with the user ID:
HTTP request node
Based on the Crawlbase docs, to scrape a Twitter user’s profile in http format, we need to complete HTTP request node in the following format:
Environment Variables. Click env - Add Variable to set the token value, so plain text will not appear in the node.
Check https://crawlbase.com/dashboard/account/docs for your crawlbase API Key.
/ , you can easily insert the API Key as a variable.
LLM node
Now, we can use LLM to analyze the result scraped by crawlbase and execute our command. The valuecontext should be body from HTTP Request node.
The following is a sample system prompt.
Test run
ClickPreview to start a test run and input twitter user id in id
Publish in the upper right corner and add it in your website.
Have fun!