Hadoop Summit San Jose 2015: YARN - Past, Present and Future

  • Published on
    16-Apr-2017

  • View
    236

  • Download
    2

Transcript

Positioning, Campaigns & 2.0 LaunchApache Hadoop YARN - 2015June 9, 2015Past, Present & FuturePage # Hortonworks Inc. 2011 2015. All Rights ReservedWe areVinod Kumar VavilapalliLong time Hadooper since 2007Apache Hadoop Committer / PMCApache MemberYahoo! -> HortonworksMapReduce -> YARN from day oneJian HeHadoop contributor since 2012Apache Hadoop Committer / PMCHortonworksAll things YARNPage # Hortonworks Inc. 2011 2015. All Rights ReservedOverviewThe Why and the WhatPage # Hortonworks Inc. 2011 2015. All Rights ReservedData architecturesTraditional architecturesSpecialized SilosPer silo security, management, governance etc.Limited ScalabilityLimited cost efficienciesFor the present and the futureHadoop repositoryCommodity storageCentralized but distributed systemScalableUniform org policy enforcementInnovation across silos!Data - HDFSCluster ResourcesPage # Hortonworks Inc. 2011 2015. All Rights ReservedResource ManagementExtracting value out of centralized data architectureA messy problemMultiple apps, frameworks, their life-cycles and evolutionTenancyI am running this system for one userIt almost never stops thereGroups, Teams, UsersSharing / isolation neededAdhoc structures get unusable real fastPage # Hortonworks Inc. 2011 2015. All Rights ReservedVaried goals & expectationsOn isolation, capacity allocations, schedulingFaster!More!Best for my clusterThroughputUtilizationElasticityService uptimeSecurityROIEverything!Right now!SLA!Page # Hortonworks Inc. 2011 2015. All Rights ReservedEnter Hadoop YARNHDFS (Scalable, Reliable Storage)YARN (Cluster Resource Management) Applications (Running Natively in Hadoop)Store all your data in one place (HDFS)Interact with that data in multiple ways (YARN Platform + Apps): Data centricScale as you go, shared, multi-tenant, secure (The Hadoop Stack)QueuesAdmins/UsersCluster ResourcesPipelinesPage # Hortonworks Inc. 2011 2015. All Rights ReservedQueues reflect org structures. Hierarchical in nature.7Hadoop YARNDistributed SystemHost of frameworks, meta-frameworks, applicationsVaried workloadsBatchInteractiveStream processingNoSQL databases.Large scaleLinear scalabilityTens of thousands of nodesMore comingPage # Hortonworks Inc. 2011 2015. All Rights ReservedPastA quick historyPage # Hortonworks Inc. 2011 2015. All Rights ReservedA brief TimelineSub-project of Apache HadoopReleases tied to Hadoop releasesAlphas and betasIn production at several large sites for MapReduce already by that timeJune-July 2010August 2011May 2012August 2013Page # Hortonworks Inc. 2011 2015. All Rights ReservedGA Releases15 October 201324 February 201407 April 201411 August 20141st GAMR binary compatibilityYARN API cleanupTesting!1st Post GABug fixesAlpha featuresRM Fail-overCS PreemptionTimeline Service V1Writable REST APIsTimeline Service V1 securityPage # Hortonworks Inc. 2011 2015. All Rights ReservedPresentPage # Hortonworks Inc. 2011 2015. All Rights ReservedLast few Hadoop releasesHadoop 2.618 November 2014Rolling UpgradesServicesNode labelsHadoop 2.721 Apr 2015Moving to JDK 7+Focus on some features next!Apache Hadoop 2.6Apache Hadoop 2.7Page # Hortonworks Inc. 2011 2015. All Rights ReservedRolling UpgradesPage # Hortonworks Inc. 2011 2015. All Rights ReservedYARN Rolling UpgradesWhy? No more losing work during upgrades!WorkflowServers first: Masters followed by per-node agentsUpgrade of Applications/Frameworks is decoupled!Work preserving RM restart: RM recovers state from NMs and appsWork preserving NM restart: NM recovers state from local diskRM fail-over is optional Page # Hortonworks Inc. 2011 2015. All Rights ReservedYARN Rolling Upgrades: A Cluster SnapshotPage # Hortonworks Inc. 2011 2015. All Rights ReservedStack Rolling UpgradesEnterprise grade rolling upgrade of a Live Hadoop ClusterJun 10, 3:25PM-4:05PMSanjay Radia & Vinod K V from HortonworksPage # Hortonworks Inc. 2011 2015. All Rights ReservedServices on YARNPage # Hortonworks Inc. 2011 2015. All Rights ReservedLong running servicesYou could run them already before 2.6!Enhancements neededLogsSecurityManagement/monitoringSharing and PlacementDiscoveryResource sharing across workload typesFault tolerance of long running servicesWork preserving AM restartAM forgetting faultsService registryPage # Hortonworks Inc. 2011 2015. All Rights ReservedProject SliderBring your existing services unmodified to YARN: slider.incubator.apache.org/HBase, Storm, Kafka already!YARNMapReduceTezStormKafkaSparkHBasePigHiveCascadingApache SliderMoreservices..DeathStar: Easy, Dynamic, Multi-tenant HBase via YARNJune 11: 1:30-2:10PMIshan Chhabra & Nitin Aggarwal from Rocket FuelAuthoring and hosting applications on YARN using SliderJun 11,11:00AM-11:40AM Sumit Mohanty & Jonathan Maron from HortonworksPage # Hortonworks Inc. 2011 2015. All Rights ReservedOperational and Developer toolingPage # Hortonworks Inc. 2011 2015. All Rights ReservedNode LabelsToday: PartitionsAdmin: I have machines of different typesImpact on capacity planning: Hey, we bought those GPU machinesTypesExclusive: This is my Precious!Non-exclusive: I get binding preference. Use it for others when idleFuture: ConstraintsTake me to a machine running JDK version 9No impact on capacity planningDefault PartitionPartition BGPUsPartition CWindowsJDK 8JDK 7JDK 7Node Labels in YARNJun 11,11:00AM-11:40AMMayank Bansal (ebay) & Wangda Tan (Hortonworks)Page # Hortonworks Inc. 2011 2015. All Rights ReservedPluggable ACLsPluggable YARN authorization modelYARN Apache Ranger integrationApache RangerQueue ACLsManagementplugin2. Submit app1. Admin manages ACLsYARNSecuring Hadoop with Apache Ranger : Strategies & Best PracticesJun 11, 3:10PM-3:50PMSelvamohan Neethiraj & Velmurugan Periasamy from HortonWorksPage # Hortonworks Inc. 2011 2015. All Rights ReservedUsabilityWhy is my application stuck?How many rack local containers did I getLots more..Why is my application stuck? What limits did it hit?What is the number of running containers of my app?How healthy is the scheduler?Page # Hortonworks Inc. 2011 2015. All Rights ReservedFuturePage # Hortonworks Inc. 2011 2015. All Rights ReservedPer-queue Policy-driven schedulingPreviouslyNowIngestionFIFOAdhocUser-fairnessAdhocFIFOIngestionFIFOCoarse policiesOne scheduling algorithm in the clusterRigidDifficult to experimentFine grained policiesOne scheduling algorithm per queueFlexibleVery easy to experiment!BatchFIFOBatchFIFOrootrootPage # Hortonworks Inc. 2011 2015. All Rights ReservedReservationsRun my workload tomorrow at 6AMNext: Persistence of the plansTimelineResources6:00AMBlock #1TimelineResources6:00AMBlock #1Block #2Reservation-based Scheduling: If Youre Late Dont Blame Us!June 10 12:05PM 12:45PMCarlo Curino & Subru Venkatraman Krishnan (Microsoft)Page # Hortonworks Inc. 2011 2015. All Rights ReservedContainerized ApplicationsRunning Containerized Applications on YARNAs a packaging mechanismAs a resource-isolation mechanismDockerAdding the notion of Container RuntimesMultiple use-casesRun my existing service on YARN via Slider + DockerRun my existing MapReduce application on YARN via a docker imageApache Hadoop YARN and the Docker EcosystemJune 9 1:45PM 2:25PMSidharta Seethana (Hortonworks) & Abin Shahab (Altiscale)Page # Hortonworks Inc. 2011 2015. All Rights ReservedDisk IsolationIsolation and scheduling dimensionsDisk CapacityIOPsBandwidthDataNodeNodeManagerMap TaskHBase RegionServerDisks on a nodeReduce TaskReadWriteLocalizationLogsShuffleReadWriteRead SpillsWrite shuffled dataRead SpillsWriteRemote IOToday: Equal allocation to all containers along all dimensionsNext: SchedulingPage # Hortonworks Inc. 2011 2015. All Rights ReservedNetwork IsolationIsolation and scheduling dimensionsIncoming bandwidthOutgoing bandwidthDataNodeNodeManagerMap TaskStorm SpoutReduce TaskWrite PipelineLocalizationLogsShuffleReadRead shuffled dataWrite outputsReadinputRemote IOToday: Equi-share Outbound bandwidthNext: SchedulingNetworkStorm BoltReadWritePage # Hortonworks Inc. 2011 2015. All Rights ReservedTimeline ServiceApplication HistoryWhere did my containers run?MapReduce specific Job History ServerNeed a generic solution beyond ResourceManager RestartCluster HistoryRun analytics on historical apps!User with most resource utilizationLargest application runRunning Applications TimelineFramework specific event collection and UIsShow me the Counters for my running MapReduce taskShow me the slowest Storm stream processing bolt while it is runningWhat exists todayA LevelDB based implementationIntegrated into MapReduce, Apache Tez, Apache HivePage # Hortonworks Inc. 2011 2015. All Rights ReservedTimeline Service 2.0Next generationTodays solution helped us understand the spaceLimited scalability and availabilityAnalyzing Hadoop Clusters is becoming a big-data problemDont want to throw away the Hadoop application metadataLarge scaleEnable near real-time analysis: Find me the user who is hammering the FileSystem with rouge applications. Now.Timeline data stored in HBase and accessible to queriesPage # Hortonworks Inc. 2011 2015. All Rights ReservedImproved UsabilityWith Timeline ServiceWhy is my application slow?Is it really slow?Why is my application failing?What happened with my application? Succeeded?Why is my cluster slow?Why is my cluster down?What happened in my clusters?Collect and use past dataTo schedule my application betterTo do better capacity planningPage # Hortonworks Inc. 2011 2015. All Rights ReservedMore..Application priorities within a queue YARN Federation 100K+ nodesNode anti-affinityDo not run two copies of my service daemon on the same machineGang schedulingRun all of my app at onceDynamic scheduling based on actual containers utilizationTime based policies10% cluster capacity for queue A from 6-9AM, but 20% from 9-12AMPrioritized queuesAdmins queue takes precedence over everything elseLot more ..HDFS on YARNGlobal schedulingUser level preemptionContainer resizingPage # Hortonworks Inc. 2011 2015. All Rights ReservedCommunityStarted with just 5 of us!104 and countingFew big contributorsAnd a long tailPage # Hortonworks Inc. 2011 2015. All Rights ReservedThank you!Page # Hortonworks Inc. 2011 2015. All Rights ReservedAddendumPage # Hortonworks Inc. 2011 2015. All Rights ReservedWork preserving ResourceManager restartResourceManager remembers some stateReconstructs the remaining from nodes and appsPage # Hortonworks Inc. 2011 2015. All Rights ReservedWork preserving NodeManager restartNodeManager remembers state on each machineReconnects to running containersPage # Hortonworks Inc. 2011 2015. All Rights ReservedResourceManager Fail-overActive/Standby based fail-overDepends on fast-recoveryPage # Hortonworks Inc. 2011 2015. All Rights Reserved