《云计算技术及应用ppt.ppt》由会员分享，可在线阅读，更多相关《云计算技术及应用ppt.ppt（39页珍藏版）》请在taowenge.com淘文阁网|工程机械CAD图纸|机械工程制图|CAD装配图下载|SolidWorks_CaTia_CAD_UG_PROE_设计图分享下载上搜索。
1、云计算技术及应用,大连理工大学计算机科学与技术学院2010年春季,基本情况,申彦明shenB810助教：齐恒clementqhB812Officehour:Fri3:30-4:30PMCoursewebsite:,教材内容,分布式系统的概况分布式与集群基本概念分布式数据库分布式文件系统GFS分布式编程MapReduce算法介绍搜索引擎与PageRank其它相关技术DataCenterBigTableAppEngine,Grading,HW：40%FinalProject:60%FinalprojectproposalProjectreports12teams,4-5students,Sylla
2、bus(Subjecttochange),Week2Mar8:Lecture1:IntroductionMar10:Lecture2:Map/ReduceTheoryandImplementation,HadoopWeek3Mar15:Lecture3&4:GuestSpeaker(8:00AM-11:35AM研教楼102)Mar17:Lecture5:DistributedFileSystemandtheGoogleFileSystemWeek4Mar22:Lecture6&7:GuestSpeaker(8:00AM-11:35AM研教楼102)Mar24:Lecture8:Distribu
3、tedGraphAlgorithmsandPageRankWeek5Mar29:Lecture9:IntroductiontoSomeProjectsMar31:Lecture10:DataCenters,Syllabus(Subjecttochange),Week6Apr5:Lecture11:SomeGoogleTechnologiesApr7:Lecture12:VirtualizationWeek7Lecture13&14:ProjectPresentationWeek8:NoclassWeek9:Lecture15&16:ProjectPresentation,GartnerRepo
4、rt,Top10StrategicTechnologyAreasfor2009VirtualizationCloudComputingServers:BeyondBladesWeb-OrientedArchitecturesEnterpriseMashupsSpecializedSystemsSocialSoftwareandSocialNetworkingUnifiedCommunicationsBusinessIntelligenceGreenInformationTechnology,Top10StrategicTechnologyAreasfor2010CloudComputingAd
5、vancedAnalyticsClientComputingITforGreenReshapingtheDataCenterSocialComputingSecurityActivityMonitoringFlashMemoryVirtualizationforAvailabilityMobileApplications,FromDesktop/HPC/GridstoInternetCloudsin30Years,HPCmovingfromcentralizedsupercomputerstogeographicallydistributeddesktops,clusters,andgrids
6、tocloudsoverlast30yearsR/DeffortsonHPC,clusters,Grids,P2P,andvirtualmachineshaslaidthefoundationofcloudcomputingthathasbeengreatlyadvocatedsince2007Locationofcomputinginfrastructureinareaswithlowercostsinhardware,software,datasets,space,andpowerrequirementsmovingfromdesktopcomputingtodatacenter-base
7、dclouds,WhatisCloudComputing?,1.Web-scaleproblems2.Largedatacenters3.Differentmodelsofcomputing4.Highly-interactiveWebapplications,1.“Web-Scale”Problems,Characteristics:Definitelydata-intensiveMayalsobeprocessingintensiveExamples:Crawling,indexing,searching,miningtheWebDatawarehousesSensornetworks“P
8、ost-genomics”lifesciencesresearchOtherscientificdata(physics,astronomy,etc.)Web2.0applications,Howmuchdata?,Googleprocesses20PBaday(2008)“allwordseverspokenbyhumanbeings”5EBCERNsLHCwillgenerate10-15PBayear,640Koughttobeenoughforanybody.,Whattodowithmoredata?,AnsweringfactoidquestionsPatternmatchingo
9、ntheWebWorksamazinglywellLearningrelationsStartwithseedinstancesSearchforpatternsontheWebUsingpatternstofindmoreinstances,HowdoImakemoney?,PetabytesofvaluablecustomerdataSittingidleinexistingdatawarehousesOverflowingoutofexistingdatawarehousesSimplybeingthrownawaySourceofdata:OLTPUserbehaviorlogsCal
10、l-centerlogsWebcrawls,publicdatasetsStructureddata(today)vs.unstructureddata(tomorrow)Howcananorganizationderivevaluefromallthisdata?,2.LargeDataCenters,Web-scaleproblems?Throwmoremachinesatit!CentralizationofresourcesinlargedatacentersNecessaryingredients:fiber,juice,andlandWhatdoOregon,Iceland,and
11、abandonedmineshaveincommon?ImportantIssues:EfficiencyRedundancyUtilizationSecurityManagementoverhead,3.DifferentComputingModels,UtilitycomputingWhybuymachineswhenyoucanrentcycles?Examples:AmazonsEC2PlatformasaService(PaaS)GivemeniceAPIandtakecareoftheimplementationExample:GoogleAppEngineSoftwareasaS
12、ervice(SaaS)Justrunitforme!Example:Gmail,“Whydoityourselfifyoucanpaysomeonetodoitforyou?”,4.WebApplications,Whatisthenatureoffuturesoftwareapplications?FromthedesktoptothebrowserSaaS=Web-basedapplicationsExamples:GoogleMaps,FacebookHowdowedeliverhighly-interactiveWeb-basedapplications?AJAX(asynchron
13、ousJavaScriptandXML)Ahackontopofamistakebuiltonsand,allheldtogetherbyducttapeandchewinggum?,SomeCloudDefinitions,IanFosteretaldefinedcloudcomputingasalarge-scaledistributedcomputingparadigm,thatisdrivenbyeconomicsofscale,inwhichapoolofabstractedvirtualized,dynamically-scalable,managedcomputingpower,
14、storage,platforms,andservicesaredeliveredondemandtoexternalcustomersovertheinternet(云计算是一种商业计算模型。它将计算任务分布在大量计算机构成的资源池上，使各种应用系统能够根据需要获取计算力、存储空间和各种软件服务。)IBMexpertsconsidercloudsthatcan:Hostavarietyofdifferentworkloads,includingbatch-stylebackendinteractive,user-facingapplicationsAllowworkloadstobedepl
15、oyedandscaled-outquicklythroughtherapidprovisioningofvirtualmachinesorphysicalmachinesSupportredundant,self-recovering,highlyscalableprogrammingmodelsthatallowworkloadstorecoverfromHW/SWfailuresMonitorresourceuseinrealtimetorebalanceallocationsondemand,InternetCloudGoals,Sharingofpeak-loadcapacityam
16、ongalargepoolofusers,improvingoverallresourceutilizationSeparationofinfrastructuremaintenancedutiesfromdomain-specificapplicationdevelopmentMajorcloudapplicationsincludeupgradedwebservices,distributeddatastorage,rawsupercomputing,andaccesstospecializedGrid,P2P,data-mining,andcontentnetworkingservice
17、s,ThreeAspectsinHardwarethatareNewinCloudComputing,Theillusionofinfinitecomputingresourcesavailableondemand,therebyeliminatingtheneedforclouduserstoplanfaraheadforprovisioningTheeliminationofanup-frontcommitmentbycloudusers,therebyallowingcompaniestostartsmallandincreasehardwareresourceswhenneededTh
18、eabilitytopaycomputingresourcesonashort-termbasisasneeded(e.g.,processorsbythehourandstoragebytheday)andreleasethemafterdoneandtherebyrewardingresourceconservation,SomeInnovativeCloudServicesandApplicationOpportunities,Smartandpervasivecloudapplicationsforindividuals,homes,communities,companies,andg
19、overnments,etc.CoordinatedCalendar,Itinerary,jobmanagement,events,andconsumerrecordmanagement(CRM)servicesCoordinatedwordprocessing,on-linepresentations,web-baseddesktops,sharingon-linedocuments,datasets,photos,video,anddatabases,etcDeployconventionalcluster,grid,P2P,socialnetworkingapplicationsincl
20、oudenvironments,morecost-effectivelyEarthboundApplicationsthatDemandElasticityandParallelismratherdatamovementCosts,OperationsinCloudComputing,UsersinteractwiththecloudtorequestserviceProvisioningtoolcarvesoutthesystemsfromthecloudconfigurationorreconfiguration,ordeprovisionTheserverscanbeeitherreal
21、orvirtualmachinesSupportingresourcesincludedistributedstoragesystem,datacenters,securitydevices,etc.,CloudComputingInstances,GoogleAmazonMicrosoftAzureIBMBlueCloud,GoogleCloudInfrastructure,Scheduler,Chubby,GFSmaster,Node,Node,Node,User,Application,Schedulerslave,GFSchunkserver,Linux,Node,MapReduceJ
22、ob,BigTableServer,GoogleCloudInfrastructure,AmazonElasticComputingCloud,SQS:SimpleQueueServiceEC2:RunningInstanceofVirtualMachinesEBS:ElasticBlockService,ProvidingtheBlockInterface,StoringVirtualMachineImagesS3:SimpleStorageService,SOAP,ObjectInterfaceSimpleDB:SimplifiedDatabase,MicrosoftAzurePlatfo
23、rm,Developer,Monitoring,ApplicationServer,ProvisioningManager,User,OpenSourceLinuxwithXen,TivoliMonitoringAgent,IBMBlueCloud,CostConsiderations:Power,Cooling,PhysicalPlant,andOperationalCosts,Costtechnologycostscostofsecurityetc.,Benefitsavailabilityopportunityconsolidationetc.,CostBreakdown,+Storag
24、e($/MByte/year)+Computing($/CPUCycles)+Networking($/bit),ResearchChallenges,ServiceavailabilityS3outage:authenticationserviceoverloadleadingtounavailabilityAppEnginepartialoutageprogrammingerrorGmail:siteunavailableSolutions:ThemanagementofaCloudComputingservicebyasinglecompanyresultsinasinglepointo
25、ffailure(SPF).IntheInternet,alargeISPusesmultiplenetworkproviderssothatfailurebyasinglecompanywillnottakethemofftheair.Similarly,weneedmultipleCloudComputingproviderstosupporteachothertoeliminateSPF.,ResearchChallenges,DataSecurityCurrentcloudofferingsareessentiallypublicratherthanprivatenetworks,ex
26、posingthesystemtomoreattackssuchasDDoSattacks.Solutions:Therearemanywellunderstoodtechnologiessuchasencryptedstorage,virtuallocalareanetworks,andnetworkmiddleboxes.,ResearchChallenges,DataTransferBottlenecksApplicationscontinuetobecomemoredata-intensive.Ifweassumeapplicationsmaybe“pulledapart”across
27、theboundariesofclouds,thismaycomplicatedataplacementandtransport.BothWANbandwidthandintra-cloudnetworkingtechnologyareperformancebottleneck.Industrialsolutions:Itisestimatedthat2/3ofthecostofWANbandwidthisconsumedbyhigh-endrouters,whereasonly1/3chargedbyfiberindustry.Wecanlowerthecostbyusingsimplerr
28、outersbuiltfromcommoditycomponentswithcentralizedcontrol,butresearchisheadingtowardsusinghigh-enddistributedrouters.,ResearchChallenges,SoftwareLicensingCurrentsoftwarelicensescommonlyrestrictthecomputersonwhichthesoftwarecanrun.Userspayforthesoftwareandthenpayanannualmaintenancefee.Manycloudcomputi
29、ngprovidersoriginallyreliedonopensourcesoftwareinpartbecausethelicensingmodelforcommercialsoftwareisnotagoodmatchtoUtilityComputing.Someideas:WecanencouragesalesforcesofsoftwarecompaniestosellproductsintoCloudComputing.Ortheycanimplementpay-per-usemodeltothesoftwaretoadapttoacloudenvironment.,Resear
30、chChallenges,ScalablestorageDifferencesbetweencommonstorageandcloudstorageThesystemisbuiltfrommanyinexpensivecommoditycomponentsthatoftenfailThesystemstoresamodestnumberoflargefilesTheworkloadsprimarilyconsistbothlargestreamingreadsandsmallrandomreads.Theworkloadsmanylarge,sequentialwritesthatappend
31、datatofilesandoncewritten,filesareseldommodifiedagain.Thecloudstorage(file)systemneedstosharemanyofthesamegoalsaspreviousdistributedfilesystemssuchasperformance,scalability,reliability,andavailability.Inaddition,itsdesignneedstobedrivenbykeyobservationsofthespecificworkloadsandtechnologicalenvironme
32、nt,bothcurrentandanticipated,thatreflectamarkeddeparturefromsomeearlierfilesystemdesignassumptions.GFSFilesaredividedintofixed-sizechunks,Chunksizeisoneofthekeydesignparameters.GFSchooses64MB,whichismuchlargerthantypicalfilesystemblocksizes.Themasterstoresthreemajortypesofmetadata:thefileandchunknam
33、espaces,themappingfromfilestochunks,andthelocationsofeachchunksreplicas.GFSsupportstheusualoperationstocreate,delete,open,close,read,andwritefiles.,ResearchChallenges,TransparentProgrammingModelProgramswrittenforcloudimplementationneedtobeautomaticallyparallelizedandexecutedonalargeclusterofcommodit
34、ymachines.Therun-timesystemshouldtakecareofthedetailsofpartitioningtheinputdata,schedulingtheprogramsexecutionacrossasetofmachines,handlingmachinefailures,andmanagingtherequiredinter-machinecommunication.Theprogrammingmodelshouldallowprogrammerswithoutmanyexperienceswithparallelanddistributedsystems
35、toeasilyutilizetheresourcesofalargedistributedsystem.MapReduceScalableDataProcessingonLargeClustersAwebprogrammingmodelimplementedforfastprocessingandgeneratinglargedatasetsAppliedmainlyinweb-scalesearchandcloudcomputingapplicationsUsersspecifyamapfunctiontogenerateasetofintermediatekey/valuepairsUs
36、ersuseareducefunctiontomergeallintermediatevalueswiththesameintermediatekey.,ResearchChallenges,SteveBallmersViewontheFutureofCloud,CloudcreatesopportunitiesandresponsibilitiesCloudlearnsandhelpsyoulearn,decideandtakeactionCloudenhancessocialandprofessionalinteractionsThecloudwantssmarterdevicesClou
37、ddrivesserveradvancesthat,inturn,drivethecloud,CloudComputingSkepticism,CLOUDCOMPUTING,Cloudcomputingissimplyabuzzwordusedtorepackagegridcomputingandutilitycomputing,bothofwhichhaveexistedfordecades.“Cloudcomputingissimplyabuzzwordusedtorepackagegridcomputingandutilitycomputing,bothofwhichhaveexiste
38、dfordecades.”,DefinitionofCloudComputing,LarryEllison,“Theinterestingthingaboutcloudcomputingisthatweveredefinedcloudcomputingtoincludeeverythingthatwealreadydo.Thecomputerindustryistheonlyindustrythatismorefashion-driventhanwomensfashion.MaybeImanidiot,butIhavenoideawhatanyoneistalkingabout.Whatisit?Itscompletegibberish.Itsinsane.Whenisthisidiocygoingtostop?”,