Ask HighScalability: Facing scaling issues with news feeds on Redis Any advice?

DEV Community — A constructive and inclusive social network for software developers. An application’s architecture must be designed primarily in accordance with principle and remain open to modification in response to statistical performance analysis. There was one big unexpected node js development technical challenge faced by the team when deploying this system that made deployments risky. Whenever a new version of the app server was deployed to production, the system would be shocked by high impulse reconnect storms originating from the data collection agents.

Every web or mobile application consists of the front end (client-side) and the back end (server-side). The latter is a combination of behind-the-scenes elements that power the user-facing side of the app. The back end consists of a server, an application, and a database.

Reliability with Unparalleled Scalability

Integrating Stream was significantly more affordable than building, maintaining and hosting an in-house solution for Bandsintown. We solve the hard scalability and integration problems, with flexible front-end components so you can do more of what you love. Access to the Ably global messaging platform requires an API key for authentication. API keys exist within the context of an Ably application and each application can have multiple API keys so that you can assign different capabilities and manage access to channels and queues.

At a certain Q&A Interest Graph based social network I used to call home, we implemented this by maintaining the „post ID“ in redis with the data itself backed-out in SimpleDb or Cassandra. You may have to go with two collections, one for news feeds (stores news, owner and the followers of news) and another one for just Users. Design a video streaming service like Youtube/Netflix where users can upload/view/search videos. The service should be scalable where a large number of users can watch and share the videos simultaneously. It will be storing and transmitting petabytes and petabytes of data. Stream has been powering their feeds for years with 100% uptime.

What do you think of the book Explain the Cloud Like I’m 10?

All major frameworks used today scale just fine — when there are scaling issues, it tends to stem from how the product was architected. Many times, a product can be overengineered, leading to scaling problems not only with the demand on a server but also in creating new features. I also design my systems to handle a specific load, which is usually some estimate of an extreme day, and behave in a deterministic way if that load is exceeded. Then, I monitor the actual performance and load in the production environment, signaling me when the latter is approaching the designed threshold.

When companies are ready to scale, part of that process means outgrowing old systems. Director of Engineering Noah Appel at INTURN, a company that turns excess inventory into capital, said they did a whole overhaul to their core SaaS product in order to implement scalability. Notably, they transitioned their front end from clunky proprietary systems to community-supported, open-source libraries and frameworks. And CTO Bernard Kowalski said home insurance marketplace Young Alfred transitioned to the cloud for infinite scalability. To provide more control over every component’s form and function, it replaces the JavaScript of Bootstrap with React.

Python vs. JavaScript – Which Language is Better?

In most cases, a full-stacker is either a former front-end developer or a back-end developer, meaning they are more proficient in one set of technologies than the other. Combining both technologies offer all these advantages, but the key benefit is that you can develop both the front-end and back-end using JavaScript. Full-stack web development with React and Node allows scaling your project to a much greater extent. Of course not, but let’s ignore that very few organizations in the world have the technological know-how to create such managed services, especially without low level control of the entire system. We have several players like Cloudflare innovating to become alternative lower-cost cloud providers. And we have k8s striving to become a cloud in a box droppable into any datacenter.

The idea is it’s practical to only have one of these for a given area. With solar power, the centralization of power generation is giving way to a more decentralized model. Would you really trust some committee or government agency to draw this line correctly? I would not because I don’t think the people who think this way actually understand how the cloud works.

What we are going to build in this article?

We need the data and scale to have a big impact in this massive industry, so scalability is critical. If we don’t think about scalability, we won’t succeed as a company. Web servers and other stateless services use immutable infrastructure and auto-scaling. When working with cloud infrastructure, we don’t troubleshoot or debug a single node. If there is a single node misbehaving, it just gets replaced. This is the “cattle, not pets” mentality made popular by Netflix.

In our newsfeed app, we would like to display the profile information of each user who is publishing a new update. Since we are not including any database and such for this tutorial, we’ll simply assign a random avatar and random name to each user who is using an instance of the application. If you are interested in implementing a Meteor publications that combines documents from different Mongo Collections I recommend the following article. This will allow you to create a publication for a news/activity feed which intermingles activities stored in multiple Mongo Collections. What are the best practices for building a scalable newsfeed using Meteor? Data science has been a hot area in software and web development for many years.

Technology

Furthermore, we leverage an expansive compute cluster for research and development. This organically encourages modular design by improving parallelism in the cluster and increasing productivity. Since we run the same software in production, those aspects of scalability carry over nicely to the live environment. Meanwhile, a collection of automated stress tests provides continual insight into performance and scaling considerations. Scalability means being able to easily handle the next busy market day.

• There are many great publications and blogs on the internet today.
• Since our back-end services operate mostly independently, we can add instances at will.
• Every web or mobile application consists of the front end (client-side) and the back end (server-side).
• Scalability in the B2B SaaS world requires the ability to onboard, support and satisfy larger companies that need to transact on larger and larger datasets.
• It’s hard to pinpoint, but the ideal time to start working on scalability is after the problems we will encounter are clear but before they overwhelm us.
• Scalability is providing a high-quality service to our ever-growing base of customers.

Replacing programming languages, retiring outdated AWS instances and adding a caching layer prepared them for traffic spikes. We’ve built a host of custom technology, which doesn’t exist for our use case or scale. But we also heavily rely on open source to avoid reinventing the wheel. During the onset of stay-at-home orders, delivery service Postmates hit unexpected growth. Manager of Engineering Sanket Agarwal said implemented code review and design processes combined with hiring the right talent helped meet customer demand while keeping essential employees safe.

How to deploy a scalable Parse LiveQuery Server?

One of the key benefits of cloud computing is its scalability, which allows users to easily and quickly scale up or down their use of computing resources, depending on their changing needs. This means that users only pay for the computing resources they actually use, rather than having to invest in expensive hardware and software upfront. This can result in significant cost savings, as well as increased flexibility and agility. There are several decent resources when it comes building activity streams – including this valuable Quora post.

This allows you to scale horizintaly using MongoDB’s easy sharding. Stay away from RDBMS‘ when using big data unless you have the need for transactions and have an endless pit of money to throw at scaling. An edge graph like Neo4J would also work well but I would recommend using mongo as you can use it for other things should you need to.

Configuring how and when to autoscale ahead of time is much easier than having to manually rescale when traffic changes drastically. It also simplifies many other aspects of maintaining a service, so it is almost always worth setting services up this way from the beginning when possible. As we build our tech, we try to keep a very strong separation of concerns throughout the system. We align the service boundaries with our teams in mind, working through scenarios with the purpose of reducing the amount of cross-team work needed for each scenario. Scale exaggerates imperfections —this is why simplicity scales. Each part of your system needs to be very well defined and understood in order to scale.