Q3 2020 Quarterly Integrator.io Product Update – Register

Q3 2020 Quarterly Integrator.io Product Update – Register2020-10-08T01:11:00+00:00

Quarterly Integrator.io Product Update

Watch Now

Matt Graney Matt Graney VP of Product
Kim Loughead Kim Loughead Head of Product Marketing

Please join us for our Q3 2020 Quarterly Integrator.io Product Update. Check out what’s new with Celigo integrator.io, see demos of our cool new features, and interact with our product team.

In this session, we have lots of exciting updates to share, including a new API Management feature called My API, our new native Snowflake connector, and enhancements to our XML parser and more!

What will you see?

  • Walk-through of My API for creating custom public endpoints
  • Introduction to Snowflake connector
  • Enhancements to XML parser
  • Ability to attach/detach for NetSuite imports

Watch Today!

Full Webinar Transcript
Good morning. Good afternoon everybody. My name is Kim Loughead. I’m VP of Product Marketing here at Celigo and welcome to our second quarterly product update. So we’re excited to show you a bunch of new stuff that we’ve added into the integrator.io platform over the past few months. I have with me today, Matt Graney, who is VP of Product Management, who is going to go through all of the updates and give you some demos and stuff. So a couple of housekeeping items. One, we are recording a session, so you will get a recording in your emails tomorrow. If you have any questions, please make use of the Q&A bar. We really wanted to be interactive, so please feel free to ask your questions. We’ll have plenty of time to get to all of them during the session. And with that, I am going to hand it off to Matt. Thank you, Kim. And good morning, good afternoon, good evening to you all, and welcome. So this is going to be the highlight, just a few major items that have gone into our releases primarily over Q3. We have, of course, other releases scheduled for the rest of the year in October and November, and then we’ll do a bit of a holiday shutdown. As always, you can check our release notes from the Knowledge Base where you’ll find a lot more details. And I’d also point you to our community, which you can also find from the Knowledge Base in the top nav bar. And I think you’ll find a lot of good content there, FAQs, and also interesting questions and answers from our customers. And increasingly, we’re even seeing some customers answering questions for other customers, which is great to see. So we’re keen to build that community and would welcome your participation. All right. So let’s get to the details here of what we’re going to cover today. The first is to introduce a new capability we called My API. Then, we’ve recently GAed a connector for Snowflake. Our timing is pretty good, I suppose, with all the news around Snowflake and their very successful IPO. We’ve made some important improvements to our XML parser that I’ll go through. And then, for those of you who are big in the NetSuite space, we’ve added support for attaching and detaching for NetSuite imports. And I will end the show with that. Okay. So now let’s get right into it, and begin by talking about My API. So for the longest time, integrator.io has been an API platform. Anything you do in the product can be done via API. Any resource you create, like an importer or exporter or a flow can be invoked via API. However, running of flow, for example, is typically asynchronous. In that, we queue the work up. It moves from stage to stage. Runs through a series of exports and imports. And so, while you can invoke it via API, the intent experience is not synchronous. So what we’ve done with My API was inspired, actually, by the need to support certain classes of applications. Now, we’ve always wanted to head in this direction. But it was prompted by a couple of applications in particular. And I think the first one, in fact, was an app called ShipWorks, which is in the sort of logistics space. And their API spec, actually, describes their product more as the client rather than the server. So typically, let’s say an app like Shopify or Google Drive you can go read their API spec and it describes what you as a client would need to do in order to interact with their service, whereas, with ShipWorks, they actually describe the patent of the API that the integration partner needs to provide, right, so the patent is reversed. And this comes up every now and again and we didn’t have a good answer for that and so we’ve introduced my API, and you’ll see it actually has a range of other capabilities too. So what it allows you to do then, as the figure shows, is for some sort of client, whether a cloud or on-prem app to make a call via HTTP to a public endpoint hosted by Integrator.io, and when calling that, it will invoke a script a javascript hook and that hook will have access to everything, the method, the parameters, the header variables, everything, and then can do whatever it needs to do and return asynchronous response, and this, of course, opens up a range of other possibilities too that we could talk about in a second. So let me just show you briefly the hook itself. For those of you who are technical, you should be able to see that this is quite similar to all the other hooks that we have. But basically, you can point your endpoint, the API endpoint to invoke a script that looks like this, and this handle request function, this is a stub, of course, will have access in the options to the HTTP method, or the headers, the query string, the body, and so on. So this will be best made real for you by showing just a really quick demo, so I’m going to switch now. If the Chrome gods behave themselves, over to integrator.io, and I have a little example here. So let me just show you first how I created this. As the account owner and in developer mode, I now have under resources this area called my API and if I were to just create a new one, you can see that all I’m asked to do is provide a name, a new endpoint, and I can choose a script, and then when I hit save it’s going to automatically generate an endpoint for me, okay. So in other words, this is now a public endpoint that anyone could hit. Now, in order to hit that, I need to provide the right token for my account, so if you go under API tokens, you can create a token so that you can authenticate when you invoke this API. But the thing is now this function I pointed it to inside the script alpha, but this function will run whenever this API is invoked. All right. So I actually prepared one a little earlier so I could show you. It’s a function called my demo for Q3 webinar. So let’s just take a look. So here’s my endpoint and it’s in a script I call a test with a certain function. If I click on the [inaudible], it will open that script and we can go down here to my demo for Q3 webinar, and all I’m actually doing in here, I am no javascript expert by any means, but I just put together a response that allows me to see exactly what values are being passed in the options field, okay. So now that I have this endpoint and we’ll just hit cancel for a sec, right. We’ve got this endpoint. I’m going to switch over to Postman, which many of you will know as a popular API client and it’s exactly this API end point is exactly what we saw. You’ll see ends in BBEB just as we had here, BBEB. So this is the real thing. And when I hit send, it’s going to run this script of mine and you see it’s providing feedback on everything that I supplied to it. Right? So it’s telling me it knows that I issued a script. It knows what all the parameters are. If I change this to post for example and hit send, it’s going to show its post. If I added another parameter here and hit send, it’s going to show that to me as well. So the point here is if I just go back to our slides– the point here is that I have access in this function to do everything that I want. Right? I can get it all done. And if you go over to our documentation, you can see some examples of what you could do in here. I haven’t repeated them all here in the interests of time. But what you’re able to do with the integrator IOAPI that you can access via this hook is to invoke other things. So for example, if you had an existing import or export, you could invoke that from that JavaScript, get it to run synchronously, get the response back, do whatever you need to do, and then return a response to the caller and do that all synchronously within a few hundred milliseconds. The system I’m running right now is our staging account but this is already available in production. In staging, of course, the performance is not as good but you’ll see it’s pretty snappy and quite responsive. So we’ve got examples in the help. Again this is already available including a nice little demo video that will maybe get into this in a little bit more detail. So again the point here is that you can run a script and get it to do whatever you want. It’s perfect for example like this where you have an application that demands a synchronous response from an API but it can be used in other things as well so perhaps maybe a response to a Webhook. If the standard response we provide is not exactly what you need and so on. There are many other cases and you’ll see that this is an area of continued investment for us so stay tuned for more features being built around this, my API capability. All right so that’s the first major piece we wanted to share. The next is Snowflake. We’ve built a connector for Snowflake and that means you can get data into and out of Snowflake as you’d expect. So on the export side, we’re providing two different ways of getting data in and on– I’m sorry. Getting data out, exporting data out of Snowflake and three different methods for getting data in, for loading data. We’re building this on top of Snowflake’s [inaudible] JS driver. So we’re using the native driver that they provide and this has proved to be quite effective already. This is relatively new. I think this came out in our September releases as GA and we’re very keen if any of you have interest in exploring this further, please let us know and we’d love to work with you directly on this. But again, this is all available today except this last one the bulk load is coming out in our October release. So I’m going to quickly go through this. If you know Snowflake, you will find this pretty familiar. If you don’t, well Snowflake is a pretty interesting product for reasons you’ll see in a second. The first is you can use Snowflake just like a sequel database effectively and just use SQL queries much like you can with our other IDBMS connectors that we have today like Postgres and MySQL. So that’s the first method. You can just use plain old SQL. In this case, select stuff from customer data, and you get a bit of some nice results. The second method that uses SQL, so still really a variant on the first method, is to support using semi-structured data, right? So you’re actually working directly with the schema, and– oh, excuse me. It means you’re able to get directly to JSON data, in this case, very quickly, right? So the syntax, of course, this is all as described by Snowflake but– so there’s two different ways of using SQL. So that’s the first way of getting data out with SQL. The second is to get data into an S3 file in a bucket using what’s called the copy-into command. I’m not going to go through this in detail, but you can see the schematic. It will give you the idea. And effectively, you’re just instructing– via Integrator.io, you’re instructing the movement of the data. So in a sense, the data doesn’t go directly through IIntergator.io because it’s going from Snowflake directly to the S3 bucket. And in this case, Integrator.io is just sort of providing the control logic if you will. Okay. And of course, that makes sense when you’re dealing with large amounts of data. So two different ways of getting the data out. And in terms of getting data in, well, not surprisingly, you could do a SQL insert. So that’s a bulk operation for that. There’s a special type here when you’re using the SQL connector in your imports. You can also work directly– there’s animation here. Sorry. You can also work directly with JSON, right? So for the Snowflake connector, there’s a special JSON data type here. So in this case, if you’ve got JSON data– in this case coming from Shopify, you could just serialize and then map it directly into this field. One of the real claims to fame of Snowflake is its ability to just work directly with JSON. And it figures it out for you, right? so you don’t necessarily have to build a static schema yourself. You just give it JSON and it figures out what to do. So there’s a lot of interesting possibilities there, not [least?] because, as many of you know, the internal language or the internal representation in Intergator.io is JSON. So this provides a pretty smooth on-ramp to getting data into Snowflake. That’s the first method. The second is using effectively updates and inserts, so not the bulk operation but more just calling a query once per page of data. I won’t spend too much time here, but you have some examples that you’ll be able to read about as well in our docs. And then the final method is sort of similar to what we saw when using S3 to get data out. And in this case, you’re going from a source, let’s say NetSuite, copying a whole lot of data, writing it to S3, and then directing the ingestion of that data from S3 into Snowflake. So it’s effectively using the [copy?] API. So three different methods to get data into Snowflake; two different ways of getting data in. So in that respect, it’s quite complete and we’re very much looking forward to continuing to engage with you all. There’s been a lot of interest so far and very keen to see more. So please do get in touch if you’re interested in exploring what we’ve done with Snowflake. Okay, the next thing relates to XML parsing. So in Integrated IO to date, the XML parser– again, as I mentioned JSON is our internal representation. So that means we try to parse XML data into JSON. And sometimes if you just have an XML document but you don’t have a schema, then it’s very difficult for a parser to know exactly what it’s meant to do with the data, what it really means, and the rules around that. So here’s a simple example with Luke Skywalker here. You’ve got these elements, and you have certain attributes against them. But it’s not clear is this meant to be name dot gender or contact dot agenda. It’s just not really clear what this is meant to be. And so what we’ve done traditionally is to be very verbose and careful. So this is what we would call a lossless translation into JSON. But it makes it quite difficult to consume downstream dealing with dollar signs and underscores and so on. It’s just extremely verbose because we don’t know. There’s not enough information to know for sure how this ought to be represented. So I’m going to switch over now to the next part of the demo to show you how we’ve changed things in recent releases. And I’m doing this from the developer playground. But of course, this is functional directly within the product as well. But I think it works well to show it here. If you haven’t already, you can get – excuse me – the developer playground when you’re in developer mode under the tools menu. So in the developer playground, you have a range of different sample documents that you can begin with. So I just chose I think by default would be this JSON record, but I just chose XML so I’ve got some data. I can then go down to the XML parser. And we’ve got auto preview turned on here, and this is exactly what was in the screenshot of my slot, right. So I don’t have any control over this. This is how it works today. But what we’ve introduced is more custom options here. So if I switch over to a custom, you’ll see now that it is doing a much better job of giving me something that looks a little bit closer to what I might expect, and I get control. So here we have attributes like ID and gender, and I’m able to make them easier to find. So for example, add a prefix here and you could see now I’ve got attribute underscore ID, attribute underscore gender so I’m able to tell very easily. And again this is the default. But we could change that to text as well right. So it gives you a bit more control over things. Let me just paste in a more complex example, and I’m going to switch back to automatic for a second because you’ll get to the point of why this is a big improvement. So here is a simple bookstore XML. I took this from the W3 schools. So our bookstore has a set of books. Each book has a title, author, year, and a price. But you can see here that it gets very, very complicated. Quite verbose takes up a lot of space of course as well. If we switch over to custom – and, of course, I still have those changes I made it – it gets a lot easier to deal with. It’s much closer automatically, oh, to what was represented here. You have other controls, and I’m not going to go into them all. But for example the resource path here. So if we just go with bookstore, we’re basically inside the bookstore element. And so now we’re just left with the array of books. So you’ll find that it will make life a bit easier for you in any existing flows you have should default to automatic because that will mean they won’t break, right. This is exactly what you’re used to. If you change this of course, because the structure changes, everything downstream will need to change. But I think you’ll find it’s a bit easier to use and ultimately will be more maintainable for you as well. So I encourage you to check that out. Okay. And let’s go back to the slides now. All right. And the last thing here relates to Netsuite imports. So in that suite, it’s, of course, possible to have relationships between objects. So for example not just to have a contact record but to show that a contact belongs to a given customer or that an opportunity belongs to a certain customer. And there wasn’t an easy way to represent that in an integrated IO, so we’ve added an option in Netsuite imports for attach and detach. So when you get to a Netsuite import you’ll see you’ve got additional options here for attach and detach. And let me just quickly show that to you as well. I’m not going to run all the way through an example but you’ll be able to find more information about that in the docs. So one second. It logged me out so I just got to sign in again. All right. So I’m not actually going to run this as a true flow but you can see the shape of things, at least, and it’ll give you a good idea of how this works. So imagine I have, say a CSV file with contacts that I’m picking up from an FTP site and those contacts– there’ll be something in the contacts that will tell me which customers they belong to. So what I could do is is grab the full set of customers. So first I’m just doing a simple lookup and I’m running a site search against customers in this simple example. And notice that when I do that I get fields like ID, and internal ID and so on. Now when I go to import contacts here, I’m attempting to do an attach. So now I’m going to take a contact and try to attach it to an existing customer. The moment I choose attach or detach what happens actually is in the mapping. I get new fields. So these fields that have locks on them, which, basically, means you can’t remove them, and in order to use this attach option for the import, I need to actually provide values for these three elements. And so for example, for the internal ID, it might be a customer internal ID. So we’re going to attach this contact to a given customer ID. For the record type, this will be hardcoded and I can say, well, yeah, this is for a customer. Okay. And so on. I’m not going to go through all of these. But the point here is that we’re now able to– when we do the import into Netsuite, we’re also able to create the relationship between objects which is obviously very important in Netsuite. And this can apply to whether it’s potentially items on an order or depending on how you’ve set things up, again, whether if you’re using Netsuite as the CRM, maybe it’s attaching opportunities to an account or a customer, contacts to a customer, and so on. So here it’s related to attach. We could also potentially break the relationships as well. So if I click detach, then you will notice my mapping changes as well. So it’s now going to show that I have to say, well what am I trying to detach this given record from since the same record could be attached to multiple things. So you can attach or detach. And this is quite important for more advanced net sweep use cases. And so we’ve added that in our September release. All right. So let’s see. All right. That was what we demoed. So the highlights, again, we went through my API, which allows you to define custom endpoints that can run a script on command and provide a synchronous response based on either all the parameters that are provided in the call, as well as, as you’ll see in in the documentation, the ability to invoke other elements of integrated IO via its API. We looked at the Snowflake connector with a couple of ways of getting data out of Snowflake and three different ways of getting data in. And we’re looking forward to hearing a lot more about that. We showed the improvements to the [XML?] pausing to make the produced [inaudible] a lot easier to read and consume downstream. And then we just talked about attach and detach for NetSuite imports. So they’re just some of the highlights. Obviously, we’ve been doing a release per month over the last six months or so. I highly encourage you to check out the release notes. Again, we have a release later this month. And one final one in November before we go quiet for the holiday period to make sure we don’t disturb your integrated IO instances. And then, in fact, our next major release after that will be in February. We’re expecting to move to quarterly releases in 2021. The feedback from you has been to slow things down a little bit. That you only want to absorb new features so quickly. So we’ll be sharing more details about our roadmaps and so on with you as needed. And, as always, we look forward to hearing your feedback. But for now I’ll hand it back to you, Kim, to see if we have any questions. [silence] Did we lose Kim? All right. Well, just in case, I’m going to go ahead and see what questions we have. Okay. We have one here related to my API and the question was whether or not it can be used for non-standard webhook responses. And, yeah, I mentioned that briefly. But you know the typical webhook response, there’s some debate actually whether it should be a 200 or a 204. But we see sometimes there are other needs, other slightly odd requirements that come from webhooks. And, yes, it means you could. If you can point your app to the endpoint, that integrated IO will generate, then your script can indeed respond any way you like, including to provide a custom response to webhooks. It’s worth adding too that right now the endpoint we generate is automatically generated and named. So you don’t have any control over the name of it right now, but clearly that’s something that we’ll be looking to to improve upon in the future. For now though, it absolutely unlocks a whole range of use cases including the ones related to integrating with certain apps. But as well as a number of the sorts of custom requirements that we hear from you. Hey, Matt. I’m back. Sorry about that. No problems. That’s all right. So we do have a few questions. So with the new attach feature, will this enable a flow to attach a PDF to a transaction in NetSuite. Ah, yes, indeed. It should be able to do that because when you do– let’s see. I think I still have this open on this import. In this case, I’m importing this as a record. But I could, I believe, just as easily– let’s imagine I was looking to transfer files, right? So this would be adding to the file cabinet. I should be able to do that. I will be able to do that actually. In this case, I would be putting it in the file cabinet. But yeah, you should also be able to attach to the transaction record. Yeah. I believe we should be able to do that. Okay, on to the custom XML parser, so how do you turn on the custom XML parser for a custom flow? Yeah. For a custom flow, it’s just a matter of– I could do it here probably if we just imagine we’re picking up files. We don’t want to do that. If we set the type to XML, then you see here, you’ve got customer automatic. For new flow, it’ll default to this new way of doing, which we call custom. Your existing flows should continue with automatic so they don’t break. But it’s as simple as that. Of course, I don’t have any sample here. You click this helper. It provides exactly the same UI that I just showed you from the developer playground. Then you may have answered this bit already. But what’s the main difference between webhooks as an import and myAPI? Yeah. So a webhook is, I would say, a signaling service almost where an application is letting you know that an event occurred. All right? Oh by the way, for example, if it’s Shopify, Shopify might say, “Oh, here’s an order.” It’s not really expecting a response other than a simple acknowledgment, typically – right? – that, yeah, we received it, whereas with myAPI, it’s more like a request, truly a request and response pattern. So that’s at a high level, the main difference. Also typically, webhooks are used at the beginning of a flow. And although the web book itself is received synchronously, and responded to synchronously, the flow downstream from that data is still kind of queued from step to step. So what you see on my screen here is, if we had a webhook, it’s the first stage here. This would happen immediately. The webhook would run, and the response would be made. But then the data would be sort of queued up and processed. And then it would move to the next stage, and it’s queued throughout. And that is the way the platform works right now with my API, though, you could expose an endpoint. And let’s say we wanted to just do this import contacts into NetSuite, we could just invoke this bubble, this import directly via its API, and the whole thing would be synchronized. So in that respect, it gives you a lot more control. You can do whatever you like there and make things, well, synchronous responses even with quite complex logic and behavior. So if you read the docs, I think, you’ll see a couple of other examples of the sorts of things you could do that are beyond really what you could do with a webhook even if there are some similarities between them. Great, thanks. Next question is, can we use the transfer feature to send contacts from NetSuite to Outlook? I think actually that would just be a regular flow, right? It would be a regular flow. Yeah. I assume if you’re talking about to the Outlook server – if you could connect to that by the API – then, yeah, you should be able to do that. I don’t think that’s dependent on anything we’ve shown here today. I mean that’s existing capability. Can the attach-detach feature in NetSuite be used with CAM? And there’s a follow-up to that, that we sometimes have had difficulties in matching customer name in CAM. So I think under the hood, there may well already be some use of this by CAM. I think from a user experience point of view, we wouldn’t immediately expose it in this way. In terms of doing the matching and so on, I know obviously, that’s one of the problems that CAM is trying to solve. Where you’re trying to go from sometimes vague information about payments and find the right open invoice to match it with. That sounds like a more complex question, though, so if you could take a note of that, I think I’d like to have probably [Previn?] from our product team get in touch with whoever was asking that one to make sure we understand that properly. Yeah, will do. So next question is if I’ve understood correctly, Snowflake will have the possibility to make a backup database from the standard database to AWS S3 bucket. Is that correct? Could you repeat the question, please? Yeah. If I’ve understood correctly, on Snowflake, we will have the possibility to make a backup database from the standard DB to AWS S3 buckets. I mean, yeah, but that too doesn’t sound specifically related to Snowflake. I mean, so what we covered here was, again, getting data out of Snowflake, right? Where it basically– maybe this is what was being referred to. So yes, in that respect, yes you could take data and get it into your own AS3 buckets sort of independent of Snowflake. If that’s the question, then yes, yes you can. Yeah, I mean Snowflake is traditionally sort of a data warehouse, so a lot of people are using it to collect data from many different sources for reporting and analytics purposes. But they’re also using it potentially as– and S3 is one of those sources, typically, if that’s where you’re maybe archiving information from from other databases but you want to bring that archived data into Snowflake for reporting and analytics. That’s the typical use case. Obviously, you can use it any way you want, but that’s typically how people are using it. So there’s one question around Form Builder which is going a bit off-topic from what today is, but we have time so we’ll throw it out there. We have noticed the Inform Builder available in the new UI for integration settings. Will there be any documentation available for how to use this? Yes. There will be. In the meantime though, I could point you to the dev playground. So back where we were on the dev playground. If you go down to say simple form and then go down to form editor, you can see quite a deal about how this works, right? And the point if– there is a document that talks about custom settings. The point of the forms, of course, is just to provide a UI to allow you to control the custom settings. So if we were to do something here. So this is a URL. Can even have some– well, this can be anything. Doesn’t matter, but if I choose some options and test form. I need a body here. Sorry I’ll just write– you can get values now, right? So there are these different fields; URL A, body, and mode. And by completing the form, your user is actually going to be supplying value. So that’s a simple example. We do have a more complete in this form field dictionary. If you then go back you can see all the different types of forms. So for example, toggling, radio buttons, multi-line fields, delimited text like here; A, B, C. All right. So you’ll see like for example here, right? So there’s quite a bit here. There’s a number of controls available. So I’d suggest you begin there and you can have dependent fields and more. So you’ll see quite a few examples in the data– I’m sorry, in the developer playground and we still owe you the docs on that. We’ve held off a little bit just because– well, we have a lot of things on our plate and it is a more advanced feature. Personally, I’d be very interested to hear more from whoever asked that question to learn about your use case. And maybe that’s something we can collaborate with you on and make sure that you’re off on the right track. So, yeah. I’d love to follow up more on that. Okay. Making another note. So the last question is back on Snowflake. So you mentioned support for AWS S3. Are there any other cloud platforms which we plan to support? Yeah. So right now, it’s S3, but we also have in mind support for Azure and Google Cloud Platform as well in the future. No committed time frame on that. But, yeah. We began with S3 and we’ll be looking at the others in coming releases. Awesome. All right. So last call, everybody. Are there any more questions from the audience? While I wait to see if anything else comes in I just want to thank Matt for putting all this material together. Hopefully, you find these sessions useful. Any feedback to make them better, more consumable, we very much want to hear it. These are sessions that are meant for you guys so we want to make sure that you’re getting as much value out of them as possible. We will have one additional one before the end of the year to cover their last release that Matt mentions. So we look forward to seeing you all again online for that session. So with that, I’m going to say thank you for your time today. Your participation in the Q&A, that was super helpful and we’ll be following up with a couple of you afterwards. And for the rest of you, I hope you enjoy the rest of your day. And if it’s not too early, have a good weekend. Thanks a lot, everybody. Thank you.