Using Google Colab a bit more with fastAPI and streamlit
Target Audience: Myself (mainly), someone who used Colab to experiment some idea and want to get some interactivity (likely frontend display to interact with)
My (failure) journey came this way (for those not interested, please skip to solution):
Originally I was tried to put in a dataset and calculate the sentence embedding using different models (LaBSE, LASER, mUSE) and compare which one perform best in my dataset.
Everything work OK, I can change the input phrase and run the cell again and again to inspect result, but I want something more interactive or render visually (thought about Tensorboard embedding pane but that’s not for my case)
(First Failure) My next idea is build a JavaScript frontend with a Docker stack and serve the result in fastAPI => suspended as all my machines are lack of enough RAM to do so (3 GB needed)
(Second Idea) Then I want to leverage Colab more and figure out streamlit can build a UI => GREAT!
I went ahead to setup streamlit in Colab, thanks to this article, it’s pretty easily setup (note that there are some package changes and need to fix a particular version for some python package, see my response in the article for detail)
(Second failure) Natively, I think streamlit can live in the same memory space and make use of the already loaded functions, but it’s not, so it basically require the streamlit python file contain all initiation of function and resources
(Third Idea) Given fastAPI was something I intended to go, so my 3rd idea is to have fastAPI serve the function I built and do request inside streamlit to access the fastAPI endpoint
To setup fastAPI following the document is easy enough.
(Obstacle) Then I try serving it with uvicon the AGSI server with uvicorn.run(…), there is a problem as this run with blocking the Colab and no other cells can run (e.g. streamlit)
(Short Victory) Afterwards, figured out a way to use multiprocessing to run uvicorn in background
(Immediate failure followed) the API, that return testing text or object are returning result, but when I go to my main endpoint which involve calling functions in Colab’s memory space (already executed), it’s always timeout, no matter how much time I set the request timeout, people also say might be “requests” is not async (which I don’t think it the root cause)
Tried to run uvicorn multiprocessing with spawn mode, not working as Tensorflow have something cannot be pickeled (I am consider new with Tensorflow, I am more a PyTorch guy)
(Finally) I realized, for fastAPI and streamlit, both of them is more capable when it’s file based (run with a self-contained .py file with all code inside)
Solution:
The target is, I want to enjoy the Colab experience that I can change a cell and run it again to check the new code. So I decided to use some magic (IPython magic), which is “%%writefile”
I restructured all the code and put thing into classes and that’s the minimum block for a cell like follow:
It would ended up preparing files per class that I can import in later classes
When I need to test code and experience, I would comment out the %%writefile magic line and execute the code in Colab memory space and check result, and change until result is satisfying, then write it into file again.
Finally, the fastAPI (uvicorn) is simple and easy with nohup call as backend task
This is the streamlit page (tunnel with ngrok):
