line·Jun 23, 2026·4 min readCurated summary

Total Capacity Exceeds 1 EB! How Do You Connect Two HDFS Systems with Different Histories? Challenges and Design Decisions in Data Platform Integration

LY Corporation’s Tech-Verse 2026 article examines how its former LINE and Yahoo Japan organizations operated HDFS platforms exceeding one exabyte in total capacity. Although both platforms used Hadoop at scale, their access models, namespace architectures, permission systems, and operational practices differed substantially. The article argues that large-scale data platforms must be designed around actual usage patterns, not just storage capacity, and previews how the two environments were later connected after organizational integration. ## Different Operating Models - Former LINE built a unified analytics environment for broad, cross-departmental data use. - Users accessed data through a web portal that managed catalogs, permissions, and role-based approval workflows rather than interacting directly with HDFS or Apache Ranger. - BI tools, reporting systems, and ETL pipelines supported diverse use cases, but integrating multiple existing clusters made operations complex. - Former Yahoo Japan evolved from a limited-purpose Hadoop deployment into a company-wide platform. - Its user interfaces and access methods were intentionally restricted, making the system easier to stabilize and support. - Yahoo Japan retained HDFS-style POSIX permissions, which limited flexibility compared with newer data-governance models. ## Different HDFS Architectures - Both platforms split their storage across multiple namespaces to overcome NameNode scaling limitations. - Each namespace used two to four NameNodes for redundancy, but the way namespaces were exposed differed: - LINE used **ViewFS**, requiring clients to maintain mount-table configurations. - Yahoo Japan used **Router-Based Federation (RBF)**, allowing routers to direct requests to the correct NameNode. - LINE shared DataNodes across namespaces, improving resource efficiency but increasing operational complexity. - LINE also had mixed NameNode and DataNode versions because several legacy platforms had been consolidated. - Yahoo Japan’s RBF design included Observer NameNodes to distribute read load. - These architectural differences affected later federation work, including connection endpoints, client configuration, network reachability, and permission management. ## Capacity and Network Challenges at LINE - Data growth exceeded forecasts, causing HDFS capacity shortages before new servers could be delivered. - Older servers were temporarily reused, leading to frequent node additions and removals. - Large changes in node count triggered HDFS Balancer activity and block redistribution, generating substantial network traffic. - Network engineers therefore had to coordinate closely with the Hadoop operations team during infrastructure changes. ## NameNode Metadata and Small-File Problems - As file and block counts increased, NameNode heap usage and processing load grew. - Larger heaps also increased garbage-collection times, making NameNodes slower and less stable. - The team analyzed regularly dumped FSImage data stored in Hive tables to identify users, paths, file counts, block counts, and data volumes. - They prioritized tables containing many small files where file compaction could significantly reduce block counts without requiring data deletion or schema changes. - File merging reduced both NameNode metadata pressure and the number of HDFS operations, improving response times for jobs. ## Namespace-Specific Load Patterns - Different namespaces experienced different types of pressure. - Temporary-file namespaces saw frequent Spark staging-file creation and deletion, producing repeated metadata updates requiring NameNode write locks. - When HDFS Balancer moved blocks, read-lock activity increased and could delay file creation and deletion. - Increasing Balancer parallelism initially worsened contention. - The team reduced parallelism to a level compatible with available DataNode disk capacity, balancing migration speed against cluster impact. ## Connecting the Two Platforms - Organizational integration introduced additional challenges beyond storage: - Determining which platform and entry point users should access - Reconciling different permission-management models - Establishing data-transfer paths between platforms - LINE’s ViewFS model depends on correctly distributed client mount tables. - Yahoo Japan’s RBF model depends on reliable, scalable, and reachable router infrastructure. - These differences directly influence cross-platform data movement, including transfers using DistCP. Large HDFS environments should be managed according to real workload behavior, namespace characteristics, and operational dependencies. Capacity planning alone is insufficient; teams should monitor metadata growth, small-file patterns, lock contention, balancing traffic, network effects, and the distinct access models of each platform.

Read original(opens in new tab)

HdfsApache SparkData PlatformApache Hadoop+3

line·Dec 5, 2025Original article

Introducing a New A/B Testing System (opens in new tab)

LY Corporation has developed an advanced A/B testing system that moves beyond simple random assignment to support dynamic user segmentation. By integrating a dedicated targeting system with a high-performance experiment assigner, the platform allows for precise experiments tailored to specific user characteristics and behaviors. This architecture enables data-driven decisions that are more relevant to localized or specialized user groups rather than relying on broad averages. ## Limitations of Traditional A/B Testing * General A/B test systems typically rely on random assignment, such as applying a hash function to a user ID (`hash(id) % 2`), which is simple and cost-effective. * While random assignment reduces selection bias, it is insufficient for hypotheses that only apply to specific cohorts, such as "iOS users living in Osaka." * Advanced systems solve this by shifting from general testing across an entire user base to personalized testing for specific segments. ## Architecture of the Targeting System * The system processes massive datasets including user information, mobile device data, and application activity stored in HDFS. * Apache Spark is used to execute complex conditional operations—such as unions, intersections, and subtractions—to refine user segments. * Segment data is written to Object Storage and then cached in Redis using a `{user_id}-{segment_id}` key format to ensure low-latency lookups during live requests. ## A/B Test Management and Assignment * The system utilizes "Central Dogma" as a configuration repository where operators and administrators define experiment parameters. * A Test Group Assigner orchestrates the process: when a client makes a request, the assigner retrieves experiment info and checks the user's segment membership in Redis. * Once a user is assigned to a specific group (e.g., Test Group 1), the system serves the corresponding content and logs the event to a data store for dashboard visualization and analysis. ## Strategic Use Cases and Future Plans * **Content Recommendation:** Testing different Machine Learning models to see which performs better for a specific user demographic. * **Targeted Incentives:** Limiting shopping discount experiments to "light users," as coupons may not significantly change the behavior of "heavy users." * **Onboarding Optimization:** Restricting UI tests to new users only, ensuring that existing users' experiences remain uninterrupted. * **Platform Expansion:** Future goals include building a unified admin interface for the entire lifecycle of an experiment and expanding the system to cover all services within LY Corporation. For organizations looking to optimize user experience, transitioning from random assignment to dynamic segmentation is essential for high-precision product development. Ensuring that segment data is cached in a high-performance store like Redis is critical to maintaining low latency when serving experimental variations in real-time.

HdfsRedisApache SparkData Architecture+3